Sep 15 12:01:59.195345 ip-10-0-140-229 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 12:01:59.603197 ip-10-0-140-229 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:01:59.603197 ip-10-0-140-229 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 12:01:59.603197 ip-10-0-140-229 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:01:59.603197 ip-10-0-140-229 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 12:01:59.603197 ip-10-0-140-229 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:01:59.606162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.606074    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 12:01:59.611277 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611261    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:01:59.611277 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611277    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611281    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611285    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611288    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611291    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611294    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611297    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611299    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611307    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611310    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611315    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611318    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611321    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611324    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611327    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611330    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611333    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611336    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611339    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611342    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:01:59.611341 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611345    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611348    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611351    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611353    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611356    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611359    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611361    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611364    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611366    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611369    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611372    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611375    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611378    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611380    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611383    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611386    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611388    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611391    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611393    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:01:59.611813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611396    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611398    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611402    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611404    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611407    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611409    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611412    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611414    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611417    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611419    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611422    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611424    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611427    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611429    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611432    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611435    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611438    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611441    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611443    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611446    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:01:59.612288 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611448    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611451    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611453    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611456    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611459    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611461    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611466    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611469    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611472    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611475    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611477    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611480    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611483    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611487    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611490    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611492    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611495    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611497    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611500    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:01:59.612790 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611502    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611504    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611507    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611510    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611512    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611515    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611517    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611520    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611524    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611526    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611529    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611531    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611534    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611536    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611539    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611541    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611544    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611546    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611549    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.611552    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:01:59.613265 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612261    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612269    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612272    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612275    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612278    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612281    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612284    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612287    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612289    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612292    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612295    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612298    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612309    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612313    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612316    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612318    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612321    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612324    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612326    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612329    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:01:59.613748 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612332    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612335    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612339    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612344    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612347    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612350    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612353    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612356    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612358    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612361    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612364    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612367    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612370    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612372    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612375    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612377    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612380    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612383    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612385    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:01:59.614285 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612388    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612390    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612394    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612397    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612400    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612402    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612405    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612407    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612410    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612412    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612415    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612417    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612420    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612424    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612426    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612429    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612432    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612434    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612437    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612439    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:01:59.614758 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612442    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612447    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612450    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612453    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612455    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612458    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612461    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612463    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612466    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612468    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612471    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612475    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612478    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612481    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612484    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612486    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612489    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612491    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612494    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:01:59.615260 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612497    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612499    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612502    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612504    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612507    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612509    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612513    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612516    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612518    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612521    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612523    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612526    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612528    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612531    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612533    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612535    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612538    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612540    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612543    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612546    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:01:59.615724 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612549    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612917    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612921    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612924    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612928    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612932    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612936    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612939    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612942    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612944    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612947    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612949    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612952    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612954    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612957    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612960    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612963    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612965    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612969    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:01:59.616237 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612972    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612974    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612977    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612980    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612982    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612985    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612988    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612990    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612993    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612996    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.612998    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613000    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613004    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613007    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613009    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613012    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613014    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613017    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613034    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613037    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:01:59.616709 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613040    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613042    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613046    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613048    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613051    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613053    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613056    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613058    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613061    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613064    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613066    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613069    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613072    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613075    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613077    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613080    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613083    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613086    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613088    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613091    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:01:59.617203 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613093    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613097    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613101    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613104    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613107    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613110    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613113    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613115    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613118    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613120    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613123    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613125    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613128    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613131    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613133    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613136    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613138    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613141    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613144    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:01:59.617679 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613147    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613149    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613152    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613155    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613159    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613163    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613166    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613168    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613171    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613173    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613176    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613179    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613181    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613184    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613186    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613189    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613191    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613194    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613196    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613199    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:01:59.618151 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613202    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613204    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613272    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613310    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613317    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613321    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613325    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613329    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613333    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613337    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613340    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613343    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613346    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613350    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613353    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613356    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613359    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613362    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613365    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613368    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613371    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613374    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613377    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613380    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 12:01:59.618643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613383    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613387    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613390    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613393    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613396    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613399    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613402    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613405    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613409    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613412    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613419    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613421    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613425    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613427    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613430    2583 flags.go:64] FLAG: --enable-server="true"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613434    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613439    2583 flags.go:64] FLAG: --event-burst="100"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613442    2583 flags.go:64] FLAG: --event-qps="50"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613445    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613448    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613451    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613455    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613457    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613461    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613464    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 15 12:01:59.619265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613467    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613470    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613473    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613476    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613479    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613482    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613485    2583 flags.go:64] FLAG: --feature-gates=""
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613491    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613494    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613498    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613501    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613504    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613507    2583 flags.go:64] FLAG: --help="false"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613510    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613513    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613516    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613526    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613529    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613533    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613536    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613539    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613542    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613545    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 12:01:59.619876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613548    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613551    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613553    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613557    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613559    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613562    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613565    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613584    2583 flags.go:64] FLAG: --lock-file=""
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613590    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613593    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613596    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613601    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613604    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613607    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613610    2583 flags.go:64] FLAG: --logging-format="text"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613613    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613618    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613621    2583 flags.go:64] FLAG: --manifest-url=""
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613623    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613628    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613631    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613635    2583 flags.go:64] FLAG: --max-pods="110"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613638    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613640    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613643    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 12:01:59.620481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613646    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613655    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613659    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613662    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613668    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613671    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613674    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613677    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613680    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613684    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613687    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613690    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613693    2583 flags.go:64] FLAG: --port="10250"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613696    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613699    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b26129ac71f78d3c"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613702    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613706    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613709    2583 flags.go:64] FLAG: --register-node="true"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613712    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613716    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613719    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613722    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613724    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613729    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613732    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 12:01:59.621337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613735    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613738    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613741    2583 flags.go:64] FLAG: --runonce="false"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613744    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613747    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613750    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613753    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613761    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613764    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613773    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613776    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613779    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613782    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613785    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613788    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613791    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613794    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613796    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613801    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613804    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613807    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613811    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613813    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613817    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613820    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 12:01:59.622407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613823    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613826    2583 flags.go:64] FLAG: --v="2"
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613829    2583 flags.go:64] FLAG: --version="false"
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613833    2583 flags.go:64] FLAG: --vmodule=""
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613837    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.613841    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613972    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613976    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613980    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613984    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613987    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613990    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613993    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613996    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.613999    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614002    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614005    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614008    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614011    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614014    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:01:59.623100 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614018    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614038    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614043    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614047    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614052    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614055    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614058    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614061    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614064    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614066    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614071    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614073    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614077    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614080    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614084    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614087    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614090    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614095    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614098    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614101    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:01:59.623602 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614104    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614106    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614109    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614111    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614114    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614117    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614119    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614122    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614124    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614127    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614130    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614133    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614136    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614138    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614141    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614143    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614146    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614149    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614151    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614154    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:01:59.624119 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614157    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614159    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614163    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614166    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614169    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614171    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614174    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614177    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614179    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614184    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614186    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614189    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614192    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614194    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614197    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614200    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614202    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614205    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614207    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:01:59.624617 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614210    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614213    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614215    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614218    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614221    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614223    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614226    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614229    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614232    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614234    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614237    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614239    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614242    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614244    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614247    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614251    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614253    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614256    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614259    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614261    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:01:59.625091 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614264    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614267    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614271    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614273    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614276    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.614278    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.614284    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.622961    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.622978    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.622998    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.623007    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623083    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623088    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623091    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:01:59.625584 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623094    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623098    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623101    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623104    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623107    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623110    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623114    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623118    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623121    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623124    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623127    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623129    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623132    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623135    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623138    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623140    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623143    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623146    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623148    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623151    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:01:59.625931 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623153    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623163    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623168    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623170    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623173    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623176    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623179    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623181    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623184    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623186    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623189    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623192    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623195    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623198    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623200    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623203    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623205    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623208    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623211    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:01:59.626438 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623213    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623216    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623219    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623221    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623224    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623227    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623230    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623232    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623235    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623237    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623240    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623242    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623245    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623248    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623251    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623254    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623256    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623259    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623262    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623265    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:01:59.626947 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623268    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623271    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623273    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623276    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623279    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623281    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623284    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623286    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623289    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623291    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623295    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623298    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623301    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623304    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623307    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623310    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623313    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623316    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623319    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623322    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:01:59.627515 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623324    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623327    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623330    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623333    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623335    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623338    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623341    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623344    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623346    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623349    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623352    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623354    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623357    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623359    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623362    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623365    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:01:59.628003 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623367    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.623372    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623508    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623514    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623518    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623520    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623525    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623529    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623532    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623535    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623538    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623542    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623545    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623548    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:01:59.628446 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623552    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623555    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623557    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623560    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623563    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623565    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623568    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623571    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623573    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623576    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623579    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623582    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623585    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623588    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623590    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623593    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623596    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623598    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623601    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623603    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:01:59.628785 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623606    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623609    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623611    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623614    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623617    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623619    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623622    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623624    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623627    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623629    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623632    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623634    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623637    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623641    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623645    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623648    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623651    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623654    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623656    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:01:59.629334 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623659    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623662    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623665    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623667    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623670    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623672    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623675    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623677    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623680    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623683    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623685    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623688    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623690    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623693    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623695    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623698    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623700    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623703    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623705    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623707    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:01:59.629784 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623710    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623713    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623715    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623717    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623720    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623722    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623725    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623728    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623731    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623733    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623736    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623738    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623741    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623743    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623746    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623748    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623750    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623753    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623756    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:01:59.630274 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623759    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623761    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623764    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623766    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623769    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623771    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623774    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623777    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:01:59.623779    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.623783    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.623884    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.627102    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.628148    2583 server.go:1019] "Starting client certificate rotation"
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.628241    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 12:01:59.630739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.628275    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 12:01:59.631105 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.631003    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 12:01:59.635146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.635129    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 12:01:59.645160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.645139    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 15 12:01:59.649006 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.648990    2583 log.go:25] "Validated CRI v1 image API"
Sep 15 12:01:59.649850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.649836    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 12:01:59.652974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.652957    2583 fs.go:135] Filesystem UUIDs: map[70883e1a-7c5d-4bbc-843d-85593f72271f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ce667a94-8516-4450-ad68-4727ce33d0fa:/dev/nvme0n1p4]
Sep 15 12:01:59.653037 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.652974    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 12:01:59.658597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.658477    2583 manager.go:216] Machine: {Timestamp:2026-09-15 12:01:59.657089802 +0000 UTC m=+0.353649390 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097824 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff4bf346fc40e13c42a525fa6cc14 SystemUUID:ec2ff4bf-346f-c40e-13c4-2a525fa6cc14 BootID:b5b5cff3-0904-4a24-bed8-5a2f5f949ba1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:1d:a2:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:1d:a2:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:be:2f:c0:da:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 12:01:59.658597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.658596    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 12:01:59.658702 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.658680    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 12:01:59.659241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.659227    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 12:01:59.659844 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.659822    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 12:01:59.659995 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.659851    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 12:01:59.660635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.660625    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 12:01:59.660667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.660638    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 12:01:59.660667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.660646    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 12:01:59.660667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.660659    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 12:01:59.662252 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.662233    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 12:01:59.665577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665565    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 12:01:59.665694 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665684    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 12:01:59.665739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665714    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 12:01:59.665739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665723    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 12:01:59.665739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665734    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 12:01:59.665818 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665742    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 15 12:01:59.665818 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.665751    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 12:01:59.666125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.666110    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 12:01:59.666125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.666128    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 12:01:59.668596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.668580    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 12:01:59.671525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.671509    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 12:01:59.671606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.671548    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 12:01:59.674509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674493    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674519    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674529    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674537    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674550    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674559    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674578    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674586    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674596    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674604    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674616    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 12:01:59.674622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.674625    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 12:01:59.675521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.675505    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 12:01:59.675521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.675521    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 12:01:59.676880 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.676858    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 12:01:59.676935 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.676861    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 12:01:59.679219 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.679208    2583 server.go:1268] "Started kubelet"
Sep 15 12:01:59.679280 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.679259    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 12:01:59.679414 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.679355    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 12:01:59.679469 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.679435    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 12:01:59.680868 ip-10-0-140-229 systemd[1]: Started Kubernetes Kubelet.
Sep 15 12:01:59.680983 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.680873    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 12:01:59.682291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.682277    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 12:01:59.685043 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.684913    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 12:01:59.688221 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.688205    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 12:01:59.689259 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.688162    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-229.ec2.internal.18d57cddf45fe8ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-229.ec2.internal,UID:ip-10-0-140-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-229.ec2.internal,},FirstTimestamp:2026-09-15 12:01:59.67918305 +0000 UTC m=+0.375742619,LastTimestamp:2026-09-15 12:01:59.67918305 +0000 UTC m=+0.375742619,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-229.ec2.internal,}"
Sep 15 12:01:59.689342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.689305    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 12:01:59.689979 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.689962    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 12:01:59.690116 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.690092    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:01:59.690750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.690701    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 12:01:59.691631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691179    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 12:01:59.691631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691196    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 12:01:59.691631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691270    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 12:01:59.691631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691278    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 12:01:59.691631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691424    2583 factory.go:55] Registering systemd factory
Sep 15 12:01:59.691631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691440    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 15 12:01:59.691919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691744    2583 factory.go:153] Registering CRI-O factory
Sep 15 12:01:59.691919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691855    2583 factory.go:223] Registration of the crio container factory successfully
Sep 15 12:01:59.691919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691906    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 12:01:59.692092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691939    2583 factory.go:103] Registering Raw factory
Sep 15 12:01:59.692092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.691966    2583 manager.go:1196] Started watching for new ooms in manager
Sep 15 12:01:59.692200 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.692175    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 12:01:59.692290 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.692254    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 12:01:59.693176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.693162    2583 manager.go:349] Starting recovery of all containers
Sep 15 12:01:59.703709 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.703494    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66jtf"
Sep 15 12:01:59.705072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.705052    2583 manager.go:354] Recovery completed
Sep 15 12:01:59.710240 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.710225    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.711237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.711221    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66jtf"
Sep 15 12:01:59.712066 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712045    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.712139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712081    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.712139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712097    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.712735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712705    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 12:01:59.712735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712724    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 12:01:59.712881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712742    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 12:01:59.712881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.712745    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.713505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.713487    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.713573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.713522    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.713573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.713535    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.714102 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.714088    2583 policy_none.go:49] "None policy: Start"
Sep 15 12:01:59.714161 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.714106    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 12:01:59.714161 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.714116    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 12:01:59.715158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.715147    2583 policy_none.go:47] "Start"
Sep 15 12:01:59.761652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.761636    2583 manager.go:337] "Starting Device Plugin manager"
Sep 15 12:01:59.761779 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.761722    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 12:01:59.761779 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.761737    2583 server.go:87] "Starting device plugin registration server"
Sep 15 12:01:59.762078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.762066    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 12:01:59.762126 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.762085    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 12:01:59.762246 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.762225    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 12:01:59.762363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.762313    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 12:01:59.762363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.762322    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 12:01:59.762937 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.762917    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 12:01:59.763011 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.762972    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:01:59.798356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.798327    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 12:01:59.799607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.799582    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 12:01:59.799723 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.799649    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 12:01:59.799723 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.799677    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 12:01:59.799811 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.799748    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 12:01:59.801649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.801633    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:01:59.862703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.862576    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.863549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.863530    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.863666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.863565    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.863666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.863579    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.863666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.863608    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.873166 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.873145    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.873261 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.873169    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-229.ec2.internal\": node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:01:59.900046 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.900004    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"]
Sep 15 12:01:59.900148 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.900077    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.901363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.901341    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.901363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.901364    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.901478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.901374    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.902333 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.902322    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.902481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.902465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.902550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.902538    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.902998 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.902982    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.903083 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.903012    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.903083 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.903043    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.903240 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.903227    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.903302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.903251    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.903302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.903261    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.904211 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.904198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.904267 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.904226    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:01:59.904836 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.904822    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:01:59.904906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.904842    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:01:59.904906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.904851    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:01:59.914824 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.914807    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:01:59.920584 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.920570    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-229.ec2.internal\" not found" node="ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.924469 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:01:59.924455    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-229.ec2.internal\" not found" node="ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.993529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.993504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c79d56c1882a8150a15d2980c27bc5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"0c79d56c1882a8150a15d2980c27bc5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.993529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.993534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c79d56c1882a8150a15d2980c27bc5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"0c79d56c1882a8150a15d2980c27bc5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:01:59.993683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:01:59.993562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23cb9ee72bd5fd0349ade4265f4e6c00-config\") pod \"kube-apiserver-proxy-ip-10-0-140-229.ec2.internal\" (UID: \"23cb9ee72bd5fd0349ade4265f4e6c00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.015478 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.015450    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.094199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.094167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c79d56c1882a8150a15d2980c27bc5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"0c79d56c1882a8150a15d2980c27bc5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.094281 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.094202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23cb9ee72bd5fd0349ade4265f4e6c00-config\") pod \"kube-apiserver-proxy-ip-10-0-140-229.ec2.internal\" (UID: \"23cb9ee72bd5fd0349ade4265f4e6c00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.094281 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.094235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c79d56c1882a8150a15d2980c27bc5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"0c79d56c1882a8150a15d2980c27bc5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.094343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.094282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c79d56c1882a8150a15d2980c27bc5c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"0c79d56c1882a8150a15d2980c27bc5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.094343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.094286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23cb9ee72bd5fd0349ade4265f4e6c00-config\") pod \"kube-apiserver-proxy-ip-10-0-140-229.ec2.internal\" (UID: \"23cb9ee72bd5fd0349ade4265f4e6c00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.094343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.094286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c79d56c1882a8150a15d2980c27bc5c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal\" (UID: \"0c79d56c1882a8150a15d2980c27bc5c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.116277 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.116230    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.217069 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.217046    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.223205 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.223185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.226565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.226549    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:00.318217 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.318177    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.419320 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.419243    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.520573 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.520546    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.524873 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.524855    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:00.621094 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.621059    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.627651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.627624    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 12:02:00.627792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.627774    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:02:00.627840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.627790    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:02:00.716593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.716517    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:56:59 +0000 UTC" deadline="2028-06-17 06:06:31.571025034 +0000 UTC"
Sep 15 12:02:00.716593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.716549    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15378h4m30.854479321s"
Sep 15 12:02:00.721931 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.721911    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.822127 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.822072    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.868368 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:00.868319    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c79d56c1882a8150a15d2980c27bc5c.slice/crio-86e61a62e407dd34dbf78e7a1ceedbddc6e45c784146048800ea6bff97a18208 WatchSource:0}: Error finding container 86e61a62e407dd34dbf78e7a1ceedbddc6e45c784146048800ea6bff97a18208: Status 404 returned error can't find the container with id 86e61a62e407dd34dbf78e7a1ceedbddc6e45c784146048800ea6bff97a18208
Sep 15 12:02:00.868833 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:00.868810    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb9ee72bd5fd0349ade4265f4e6c00.slice/crio-f6c40fa219b9ceee166148d7e122a43ce93a54ce5b5da81f70cb2e35c8a1a10e WatchSource:0}: Error finding container f6c40fa219b9ceee166148d7e122a43ce93a54ce5b5da81f70cb2e35c8a1a10e: Status 404 returned error can't find the container with id f6c40fa219b9ceee166148d7e122a43ce93a54ce5b5da81f70cb2e35c8a1a10e
Sep 15 12:02:00.875663 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.875646    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:02:00.893844 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.893817    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:00.922241 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:00.922206    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-229.ec2.internal\" not found"
Sep 15 12:02:00.967989 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.967927    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:00.991206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:00.991160    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:01.005760 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.005734    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 12:02:01.007017 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.007000    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal"
Sep 15 12:02:01.016294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.016276    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 12:02:01.604698 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.604656    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:01.666658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.666617    2583 apiserver.go:52] "Watching apiserver"
Sep 15 12:02:01.673308 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.673281    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 12:02:01.673927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.673904    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9qdwb","openshift-multus/multus-lct5p","kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal","openshift-multus/network-metrics-daemon-g2n6z","openshift-network-diagnostics/network-check-target-mjfj7","openshift-network-operator/iptables-alerter-wcpvh","openshift-ovn-kubernetes/ovnkube-node-vrzkf","kube-system/global-pull-secret-syncer-66f9b","kube-system/konnectivity-agent-bxmpx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88","openshift-cluster-node-tuning-operator/tuned-rlnwn","openshift-dns/node-resolver-xw4k7","openshift-image-registry/node-ca-f79s4"]
Sep 15 12:02:01.675413 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.675390    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.676722 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.676675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.677959 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.677935    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 12:02:01.678320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.678273    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 12:02:01.678861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.678618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.678861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.678780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n2b8l\""
Sep 15 12:02:01.679284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.679267    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.679284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.679279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.679677 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.679571    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bzc5t\""
Sep 15 12:02:01.679767 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.679731    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.680504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.680486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:01.680598 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.680559    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:01.681311 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.681292    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.681974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.681860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:01.681974 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.681940    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:01.683527 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.683244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.683527 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.683311    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:01.684770 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.684534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.685684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 12:02:01.685684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685519    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.685684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685565    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbhtj\""
Sep 15 12:02:01.685684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685616    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 12:02:01.685684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685473    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b6dc2\""
Sep 15 12:02:01.686080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685691    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.686080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685872    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 12:02:01.686080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.685908    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.686247 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.686131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 12:02:01.686373 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.686353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.686869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.686835    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 12:02:01.687022 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.687002    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tftdz\""
Sep 15 12:02:01.687375 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.687355    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.687461 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.687385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 12:02:01.687461 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.687444    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.688415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.688392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.689019 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.688995    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 12:02:01.689940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.689240    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.689940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.689260    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.689940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.689800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wrvpq\""
Sep 15 12:02:01.689940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.689925    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 12:02:01.690830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.690808    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 12:02:01.692792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.692740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sf4bj\""
Sep 15 12:02:01.692941 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.692901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 12:02:01.692941 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.692903    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 12:02:01.694182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.693182    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.695274 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.695253    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.696478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.696457    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.696943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.696743    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nhc29\""
Sep 15 12:02:01.696943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.696766    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.697576 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.697552    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 12:02:01.697762 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.697698    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:01.697849 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.697779    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:01.697909 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.697698    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7wff7\""
Sep 15 12:02:01.703013 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.702984    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 12:02:01.708393 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/144046aa-3d7c-4925-8259-da5480d92d3e-hosts-file\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.708498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144046aa-3d7c-4925-8259-da5480d92d3e-tmp-dir\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.708498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-host-slash\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.708498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-dbus\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.708664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncnj\" (UniqueName: \"kubernetes.io/projected/c942e6b8-5d26-4666-8a62-5fdf72686e63-kube-api-access-9ncnj\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.708664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-kubelet\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.708664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.708664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58n5\" (UniqueName: \"kubernetes.io/projected/15c02746-890f-4bfa-9bbe-c20980af3d45-kube-api-access-x58n5\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.708664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-log-socket\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.708664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-env-overrides\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-kubelet-config\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-system-cni-dir\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea5ba13e-d405-4fb7-9030-811b949df8d8-host\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea5ba13e-d405-4fb7-9030-811b949df8d8-serviceca\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnczp\" (UniqueName: \"kubernetes.io/projected/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-kube-api-access-qnczp\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-ovn\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-netns\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.708962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-registration-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-system-cni-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.708995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f91cc570-5cdc-41a7-8076-28d2370630f7-cni-binary-copy\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zzn\" (UniqueName: \"kubernetes.io/projected/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-kube-api-access-x4zzn\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-systemd-units\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-cni-multus\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-hostroot\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-multus-certs\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-etc-kubernetes\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq55w\" (UniqueName: \"kubernetes.io/projected/ea5ba13e-d405-4fb7-9030-811b949df8d8-kube-api-access-pq55w\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-slash\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-daemon-config\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-cnibin\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-cni-binary-copy\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-kubelet\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.709468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-cni-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-cnibin\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-socket-dir-parent\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-k8s-cni-cncf-io\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-socket-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovnkube-config\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovn-node-metrics-cert\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovnkube-script-lib\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.709976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-sys-fs\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-var-lib-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-conf-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gfkz\" (UniqueName: \"kubernetes.io/projected/144046aa-3d7c-4925-8259-da5480d92d3e-kube-api-access-9gfkz\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.710156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-iptables-alerter-script\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tr4x\" (UniqueName: \"kubernetes.io/projected/c57ce7a1-dcfc-4485-8251-2f68887d51ac-kube-api-access-2tr4x\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9gv\" (UniqueName: \"kubernetes.io/projected/f91cc570-5cdc-41a7-8076-28d2370630f7-kube-api-access-df9gv\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca52c69a-01eb-448e-86c0-7a8018d39ad8-konnectivity-ca\") pod \"konnectivity-agent-bxmpx\" (UID: \"ca52c69a-01eb-448e-86c0-7a8018d39ad8\") " pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-systemd\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-etc-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-node-log\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-cni-bin\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-cni-netd\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-cni-bin\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca52c69a-01eb-448e-86c0-7a8018d39ad8-agent-certs\") pod \"konnectivity-agent-bxmpx\" (UID: \"ca52c69a-01eb-448e-86c0-7a8018d39ad8\") " pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-device-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-run-netns\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.710897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.710624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.722961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.722911    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gdq6h"
Sep 15 12:02:01.732724 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.732694    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gdq6h"
Sep 15 12:02:01.791795 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.791770    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 12:02:01.804956 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.804908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"0c79d56c1882a8150a15d2980c27bc5c","Type":"ContainerStarted","Data":"86e61a62e407dd34dbf78e7a1ceedbddc6e45c784146048800ea6bff97a18208"}
Sep 15 12:02:01.806234 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.806211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal" event={"ID":"23cb9ee72bd5fd0349ade4265f4e6c00","Type":"ContainerStarted","Data":"f6c40fa219b9ceee166148d7e122a43ce93a54ce5b5da81f70cb2e35c8a1a10e"}
Sep 15 12:02:01.810789 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-sys-fs\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.810893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-var-lib-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.810893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-conf-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.810893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gfkz\" (UniqueName: \"kubernetes.io/projected/144046aa-3d7c-4925-8259-da5480d92d3e-kube-api-access-9gfkz\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-modprobe-d\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-sys-fs\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-var-lib-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-conf-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-iptables-alerter-script\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tr4x\" (UniqueName: \"kubernetes.io/projected/c57ce7a1-dcfc-4485-8251-2f68887d51ac-kube-api-access-2tr4x\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.810996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9gv\" (UniqueName: \"kubernetes.io/projected/f91cc570-5cdc-41a7-8076-28d2370630f7-kube-api-access-df9gv\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca52c69a-01eb-448e-86c0-7a8018d39ad8-konnectivity-ca\") pod \"konnectivity-agent-bxmpx\" (UID: \"ca52c69a-01eb-448e-86c0-7a8018d39ad8\") " pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.811078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-systemd\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-etc-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-node-log\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-cni-bin\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-cni-netd\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-etc-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-systemd\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-cni-bin\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-cni-netd\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-node-log\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-cni-bin\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca52c69a-01eb-448e-86c0-7a8018d39ad8-agent-certs\") pod \"konnectivity-agent-bxmpx\" (UID: \"ca52c69a-01eb-448e-86c0-7a8018d39ad8\") " pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-device-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.811462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-run-netns\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/144046aa-3d7c-4925-8259-da5480d92d3e-hosts-file\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144046aa-3d7c-4925-8259-da5480d92d3e-tmp-dir\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysctl-conf\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811587    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-iptables-alerter-script\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-host-slash\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-dbus\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-lib-modules\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-run-netns\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ca52c69a-01eb-448e-86c0-7a8018d39ad8-konnectivity-ca\") pod \"konnectivity-agent-bxmpx\" (UID: \"ca52c69a-01eb-448e-86c0-7a8018d39ad8\") " pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncnj\" (UniqueName: \"kubernetes.io/projected/c942e6b8-5d26-4666-8a62-5fdf72686e63-kube-api-access-9ncnj\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-kubelet\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-device-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58n5\" (UniqueName: \"kubernetes.io/projected/15c02746-890f-4bfa-9bbe-c20980af3d45-kube-api-access-x58n5\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.811961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811798    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-host-slash\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-host\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-cni-bin\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-dbus\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-tuned\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-log-socket\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-env-overrides\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/144046aa-3d7c-4925-8259-da5480d92d3e-hosts-file\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-kubelet-config\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-system-cni-dir\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.811936    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea5ba13e-d405-4fb7-9030-811b949df8d8-host\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.812019    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:02.311980087 +0000 UTC m=+3.008539644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-kubelet\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144046aa-3d7c-4925-8259-da5480d92d3e-tmp-dir\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.811939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea5ba13e-d405-4fb7-9030-811b949df8d8-host\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-kubelet-config\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:01.812786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea5ba13e-d405-4fb7-9030-811b949df8d8-serviceca\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-kubernetes\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysctl-d\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-system-cni-dir\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-log-socket\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-env-overrides\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.812983    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnczp\" (UniqueName: \"kubernetes.io/projected/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-kube-api-access-qnczp\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-ovn\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-ovn\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-netns\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813564    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea5ba13e-d405-4fb7-9030-811b949df8d8-serviceca\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-netns\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.813614 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-sys\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2sr\" (UniqueName: \"kubernetes.io/projected/20da350e-23cd-4bfa-8e7c-783b11bb435c-kube-api-access-lm2sr\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.813734    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-registration-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.813801    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:02.31378142 +0000 UTC m=+3.010340984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-registration-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-system-cni-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.813931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f91cc570-5cdc-41a7-8076-28d2370630f7-cni-binary-copy\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-system-cni-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zzn\" (UniqueName: \"kubernetes.io/projected/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-kube-api-access-x4zzn\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-systemd-units\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-cni-multus\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-hostroot\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.814299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-multus-certs\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-etc-kubernetes\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq55w\" (UniqueName: \"kubernetes.io/projected/ea5ba13e-d405-4fb7-9030-811b949df8d8-kube-api-access-pq55w\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysconfig\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-systemd\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-slash\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-daemon-config\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-cnibin\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-hostroot\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-cni-binary-copy\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-var-lib-kubelet\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-kubelet\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-cni-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-cnibin\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-socket-dir-parent\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-k8s-cni-cncf-io\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.814949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-run\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20da350e-23cd-4bfa-8e7c-783b11bb435c-tmp\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-socket-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovnkube-config\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-daemon-config\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovn-node-metrics-cert\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovnkube-script-lib\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f91cc570-5cdc-41a7-8076-28d2370630f7-cni-binary-copy\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-kubelet\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-etc-kubernetes\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15c02746-890f-4bfa-9bbe-c20980af3d45-cnibin\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-run-openvswitch\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.815934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-multus-certs\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-host-slash\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-cni-dir\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-run-k8s-cni-cncf-io\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-host-var-lib-cni-multus\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-cnibin\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.815872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-cni-binary-copy\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15c02746-890f-4bfa-9bbe-c20980af3d45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovnkube-config\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f91cc570-5cdc-41a7-8076-28d2370630f7-multus-socket-dir-parent\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c942e6b8-5d26-4666-8a62-5fdf72686e63-systemd-units\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovnkube-script-lib\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.816858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.816649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c57ce7a1-dcfc-4485-8251-2f68887d51ac-socket-dir\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.821462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.821138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ca52c69a-01eb-448e-86c0-7a8018d39ad8-agent-certs\") pod \"konnectivity-agent-bxmpx\" (UID: \"ca52c69a-01eb-448e-86c0-7a8018d39ad8\") " pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.821462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.821145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tr4x\" (UniqueName: \"kubernetes.io/projected/c57ce7a1-dcfc-4485-8251-2f68887d51ac-kube-api-access-2tr4x\") pod \"aws-ebs-csi-driver-node-g6c88\" (UID: \"c57ce7a1-dcfc-4485-8251-2f68887d51ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:01.821462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.821294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gfkz\" (UniqueName: \"kubernetes.io/projected/144046aa-3d7c-4925-8259-da5480d92d3e-kube-api-access-9gfkz\") pod \"node-resolver-xw4k7\" (UID: \"144046aa-3d7c-4925-8259-da5480d92d3e\") " pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:01.821462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.821459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncnj\" (UniqueName: \"kubernetes.io/projected/c942e6b8-5d26-4666-8a62-5fdf72686e63-kube-api-access-9ncnj\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.822424 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.821547    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:01.822424 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.821573    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:01.822424 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.821592    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:01.822424 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:01.821917    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:02:02.32186618 +0000 UTC m=+3.018425760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:01.822424 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.822054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9gv\" (UniqueName: \"kubernetes.io/projected/f91cc570-5cdc-41a7-8076-28d2370630f7-kube-api-access-df9gv\") pod \"multus-lct5p\" (UID: \"f91cc570-5cdc-41a7-8076-28d2370630f7\") " pod="openshift-multus/multus-lct5p"
Sep 15 12:02:01.823030 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.822999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58n5\" (UniqueName: \"kubernetes.io/projected/15c02746-890f-4bfa-9bbe-c20980af3d45-kube-api-access-x58n5\") pod \"multus-additional-cni-plugins-9qdwb\" (UID: \"15c02746-890f-4bfa-9bbe-c20980af3d45\") " pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:01.823139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.823002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c942e6b8-5d26-4666-8a62-5fdf72686e63-ovn-node-metrics-cert\") pod \"ovnkube-node-vrzkf\" (UID: \"c942e6b8-5d26-4666-8a62-5fdf72686e63\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:01.823784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.823764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnczp\" (UniqueName: \"kubernetes.io/projected/54ab5368-9c3b-4dbe-8740-e89279ab3c6e-kube-api-access-qnczp\") pod \"iptables-alerter-wcpvh\" (UID: \"54ab5368-9c3b-4dbe-8740-e89279ab3c6e\") " pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:01.823869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.823840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq55w\" (UniqueName: \"kubernetes.io/projected/ea5ba13e-d405-4fb7-9030-811b949df8d8-kube-api-access-pq55w\") pod \"node-ca-f79s4\" (UID: \"ea5ba13e-d405-4fb7-9030-811b949df8d8\") " pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:01.825240 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.825205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zzn\" (UniqueName: \"kubernetes.io/projected/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-kube-api-access-x4zzn\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:01.916577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-modprobe-d\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysctl-conf\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-lib-modules\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-host\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-tuned\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-kubernetes\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916706    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-modprobe-d\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-host\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916706    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysctl-conf\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysctl-d\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916785    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-kubernetes\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.916808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-lib-modules\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysctl-d\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-sys\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2sr\" (UniqueName: \"kubernetes.io/projected/20da350e-23cd-4bfa-8e7c-783b11bb435c-kube-api-access-lm2sr\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysconfig\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-systemd\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-sys\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-var-lib-kubelet\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-run\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-sysconfig\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20da350e-23cd-4bfa-8e7c-783b11bb435c-tmp\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-var-lib-kubelet\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.916970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-systemd\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.917338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.917011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/20da350e-23cd-4bfa-8e7c-783b11bb435c-run\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.920184 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.920078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/20da350e-23cd-4bfa-8e7c-783b11bb435c-etc-tuned\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.920184 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.920155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20da350e-23cd-4bfa-8e7c-783b11bb435c-tmp\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.924930 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.924905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2sr\" (UniqueName: \"kubernetes.io/projected/20da350e-23cd-4bfa-8e7c-783b11bb435c-kube-api-access-lm2sr\") pod \"tuned-rlnwn\" (UID: \"20da350e-23cd-4bfa-8e7c-783b11bb435c\") " pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:01.988998 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.988959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:01.999307 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:01.999279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xw4k7"
Sep 15 12:02:02.010138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.010112    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f79s4"
Sep 15 12:02:02.018027 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.018005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:02.027983 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.027961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wcpvh"
Sep 15 12:02:02.036747 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.036726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lct5p"
Sep 15 12:02:02.045318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.045294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9qdwb"
Sep 15 12:02:02.052884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.052866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88"
Sep 15 12:02:02.060629 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.060610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rlnwn"
Sep 15 12:02:02.321081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.321002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:02.321081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.321075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:02.321296 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.321169    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:02.321296 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.321242    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:03.321221138 +0000 UTC m=+4.017780700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:02.321296 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.321178    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:02.321455 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.321333    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:03.321313021 +0000 UTC m=+4.017872597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:02.421438 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.421406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:02.421612 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.421591    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:02.421651 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.421615    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:02.421651 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.421626    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:02.421752 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.421686    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:02:03.421666266 +0000 UTC m=+4.118225836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:02.616394 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.616179    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91cc570_5cdc_41a7_8076_28d2370630f7.slice/crio-fe4f0828f19458a058d316e4a26e93377ccf7cff78f1e3fd93f3ca23fd471133 WatchSource:0}: Error finding container fe4f0828f19458a058d316e4a26e93377ccf7cff78f1e3fd93f3ca23fd471133: Status 404 returned error can't find the container with id fe4f0828f19458a058d316e4a26e93377ccf7cff78f1e3fd93f3ca23fd471133
Sep 15 12:02:02.620132 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.620103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5ba13e_d405_4fb7_9030_811b949df8d8.slice/crio-d81a8043cf26fd31a2fef438486bc2d3d95953f3d74a436a6b1adf4634f85659 WatchSource:0}: Error finding container d81a8043cf26fd31a2fef438486bc2d3d95953f3d74a436a6b1adf4634f85659: Status 404 returned error can't find the container with id d81a8043cf26fd31a2fef438486bc2d3d95953f3d74a436a6b1adf4634f85659
Sep 15 12:02:02.622580 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.622557    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da350e_23cd_4bfa_8e7c_783b11bb435c.slice/crio-4cc4dfdc80d8551bc3bcf77ec4c2f1bb7da0b515bb883fcfa4a23b2ce7518705 WatchSource:0}: Error finding container 4cc4dfdc80d8551bc3bcf77ec4c2f1bb7da0b515bb883fcfa4a23b2ce7518705: Status 404 returned error can't find the container with id 4cc4dfdc80d8551bc3bcf77ec4c2f1bb7da0b515bb883fcfa4a23b2ce7518705
Sep 15 12:02:02.623387 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.623355    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57ce7a1_dcfc_4485_8251_2f68887d51ac.slice/crio-ffae6cb062953ed813fd314d6559be4d14455cce17a056e50b236d2ea1b65052 WatchSource:0}: Error finding container ffae6cb062953ed813fd314d6559be4d14455cce17a056e50b236d2ea1b65052: Status 404 returned error can't find the container with id ffae6cb062953ed813fd314d6559be4d14455cce17a056e50b236d2ea1b65052
Sep 15 12:02:02.624469 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.624445    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc942e6b8_5d26_4666_8a62_5fdf72686e63.slice/crio-60781c48de6b59a62877436297ac1911a28f4a8198c36724e1459ebd636b6331 WatchSource:0}: Error finding container 60781c48de6b59a62877436297ac1911a28f4a8198c36724e1459ebd636b6331: Status 404 returned error can't find the container with id 60781c48de6b59a62877436297ac1911a28f4a8198c36724e1459ebd636b6331
Sep 15 12:02:02.625184 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.625160    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ab5368_9c3b_4dbe_8740_e89279ab3c6e.slice/crio-6c238472c6488b45e7ca0f22a0e4e8c7072439f0d16015327890f966c243e39f WatchSource:0}: Error finding container 6c238472c6488b45e7ca0f22a0e4e8c7072439f0d16015327890f966c243e39f: Status 404 returned error can't find the container with id 6c238472c6488b45e7ca0f22a0e4e8c7072439f0d16015327890f966c243e39f
Sep 15 12:02:02.626564 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.626534    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c02746_890f_4bfa_9bbe_c20980af3d45.slice/crio-aaaf7505887be3442f8cf7fcc8df241b593df6ac45d67e19549462b55ffb3cc7 WatchSource:0}: Error finding container aaaf7505887be3442f8cf7fcc8df241b593df6ac45d67e19549462b55ffb3cc7: Status 404 returned error can't find the container with id aaaf7505887be3442f8cf7fcc8df241b593df6ac45d67e19549462b55ffb3cc7
Sep 15 12:02:02.627917 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.627534    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144046aa_3d7c_4925_8259_da5480d92d3e.slice/crio-4e64af2ab3f9ab1baf18bb23d1b7bc0f594cfa0885677e09988c1a58312acabb WatchSource:0}: Error finding container 4e64af2ab3f9ab1baf18bb23d1b7bc0f594cfa0885677e09988c1a58312acabb: Status 404 returned error can't find the container with id 4e64af2ab3f9ab1baf18bb23d1b7bc0f594cfa0885677e09988c1a58312acabb
Sep 15 12:02:02.628430 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:02.628389    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca52c69a_01eb_448e_86c0_7a8018d39ad8.slice/crio-c19a5de38ef53ad989c425ef1c690e83939f81b5154ae2e43f1dd9fe3804ba31 WatchSource:0}: Error finding container c19a5de38ef53ad989c425ef1c690e83939f81b5154ae2e43f1dd9fe3804ba31: Status 404 returned error can't find the container with id c19a5de38ef53ad989c425ef1c690e83939f81b5154ae2e43f1dd9fe3804ba31
Sep 15 12:02:02.734650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.734614    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:57:01 +0000 UTC" deadline="2028-03-21 05:15:37.618297135 +0000 UTC"
Sep 15 12:02:02.734650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.734645    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13265h13m34.883655192s"
Sep 15 12:02:02.800404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.800367    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:02.800540 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:02.800504    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:02.811291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.811260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal" event={"ID":"23cb9ee72bd5fd0349ade4265f4e6c00","Type":"ContainerStarted","Data":"cfba1ce108bdcb594b590464f077a2e8cb524619d107e8bf997712ff379c5ade"}
Sep 15 12:02:02.812334 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.812310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xw4k7" event={"ID":"144046aa-3d7c-4925-8259-da5480d92d3e","Type":"ContainerStarted","Data":"4e64af2ab3f9ab1baf18bb23d1b7bc0f594cfa0885677e09988c1a58312acabb"}
Sep 15 12:02:02.813361 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.813340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerStarted","Data":"aaaf7505887be3442f8cf7fcc8df241b593df6ac45d67e19549462b55ffb3cc7"}
Sep 15 12:02:02.814361 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.814337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wcpvh" event={"ID":"54ab5368-9c3b-4dbe-8740-e89279ab3c6e","Type":"ContainerStarted","Data":"6c238472c6488b45e7ca0f22a0e4e8c7072439f0d16015327890f966c243e39f"}
Sep 15 12:02:02.815458 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.815442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"60781c48de6b59a62877436297ac1911a28f4a8198c36724e1459ebd636b6331"}
Sep 15 12:02:02.816524 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.816498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlnwn" event={"ID":"20da350e-23cd-4bfa-8e7c-783b11bb435c","Type":"ContainerStarted","Data":"4cc4dfdc80d8551bc3bcf77ec4c2f1bb7da0b515bb883fcfa4a23b2ce7518705"}
Sep 15 12:02:02.817532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.817514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88" event={"ID":"c57ce7a1-dcfc-4485-8251-2f68887d51ac","Type":"ContainerStarted","Data":"ffae6cb062953ed813fd314d6559be4d14455cce17a056e50b236d2ea1b65052"}
Sep 15 12:02:02.818534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.818516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f79s4" event={"ID":"ea5ba13e-d405-4fb7-9030-811b949df8d8","Type":"ContainerStarted","Data":"d81a8043cf26fd31a2fef438486bc2d3d95953f3d74a436a6b1adf4634f85659"}
Sep 15 12:02:02.819506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.819490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxmpx" event={"ID":"ca52c69a-01eb-448e-86c0-7a8018d39ad8","Type":"ContainerStarted","Data":"c19a5de38ef53ad989c425ef1c690e83939f81b5154ae2e43f1dd9fe3804ba31"}
Sep 15 12:02:02.820439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.820413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lct5p" event={"ID":"f91cc570-5cdc-41a7-8076-28d2370630f7","Type":"ContainerStarted","Data":"fe4f0828f19458a058d316e4a26e93377ccf7cff78f1e3fd93f3ca23fd471133"}
Sep 15 12:02:02.826513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.826466    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-229.ec2.internal" podStartSLOduration=1.826448755 podStartE2EDuration="1.826448755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:02.825449381 +0000 UTC m=+3.522008979" watchObservedRunningTime="2026-09-15 12:02:02.826448755 +0000 UTC m=+3.523008335"
Sep 15 12:02:02.839811 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:02.839794    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:02:03.328487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.328433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:03.328684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.328519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:03.328684 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.328669    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:03.328789 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.328731    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:05.328712806 +0000 UTC m=+6.025272370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:03.329159 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.329140    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:03.329256 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.329198    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:05.329182145 +0000 UTC m=+6.025741725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:03.429405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.429361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:03.429670 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.429625    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:03.429670 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.429651    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:03.429670 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.429664    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:03.429837 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.429720    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:02:05.42970263 +0000 UTC m=+6.126262201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:03.735287 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.735171    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:57:01 +0000 UTC" deadline="2028-07-03 06:17:52.880149788 +0000 UTC"
Sep 15 12:02:03.735287 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.735235    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15762h15m49.144918736s"
Sep 15 12:02:03.803752 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.803118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:03.803752 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.803237    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:03.803752 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:03.803311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:03.803752 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:03.803381    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:04.800964 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:04.800884    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:04.801575 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:04.801044    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:05.345045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:05.344993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:05.345253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:05.345087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:05.345326 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.345254    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:05.345326 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.345317    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:09.345299891 +0000 UTC m=+10.041859452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:05.345732 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.345702    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:05.345845 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.345753    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:09.345738261 +0000 UTC m=+10.042297823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:05.446214 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:05.446081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:05.446405 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.446271    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:05.446405 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.446299    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:05.446405 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.446313    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:05.446405 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.446372    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:02:09.446352681 +0000 UTC m=+10.142912241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:05.800769 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:05.800692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:05.800769 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:05.800719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:05.800982 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.800831    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:05.801342 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:05.801016    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:06.800044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:06.799997    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:06.800237 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:06.800151    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:07.800306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:07.800272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:07.800761 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:07.800398    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:07.800761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:07.800457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:07.800761 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:07.800546    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:08.800804 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:08.800729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:08.801290 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:08.800924    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:09.378839 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:09.378798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:09.379050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:09.378860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:09.379050 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.378989    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:09.379171 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.379066    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:17.379047008 +0000 UTC m=+18.075606568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:09.379456 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.379439    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:09.379530 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.379489    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:17.379475211 +0000 UTC m=+18.076034772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:09.480084 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:09.480032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:09.480275 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.480205    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:09.480275 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.480233    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:09.480275 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.480246    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:09.480431 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.480313    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:02:17.480293519 +0000 UTC m=+18.176853076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:09.801422 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:09.801384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:09.801850 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.801505    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:09.801850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:09.801559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:09.801850 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:09.801664    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:10.800397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:10.800361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:10.800603 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:10.800515    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:11.799993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:11.799959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:11.800476 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:11.800082    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:11.800476 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:11.800180    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:11.800476 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:11.800274    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:12.800507 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:12.800477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:12.801073 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:12.800605    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:13.800191 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:13.800150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:13.800363 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:13.800279    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:13.800363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:13.800157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:13.800492 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:13.800367    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:14.800723 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:14.800685    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:14.801238 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:14.800827    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:15.800264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:15.800222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:15.800264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:15.800266    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:15.800543 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:15.800372    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:15.800543 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:15.800453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:16.800645 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:16.800611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:16.801003 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:16.800724    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:17.439664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:17.439619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:17.439838 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:17.439712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:17.439838 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.439790    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:17.439838 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.439809    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:17.439988 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.439865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:33.439847609 +0000 UTC m=+34.136407166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:17.439988 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.439883    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:33.439875955 +0000 UTC m=+34.136435511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:17.540222 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:17.540182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:17.540401 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.540335    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:17.540401 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.540355    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:17.540401 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.540367    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:17.540569 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.540433    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:02:33.540415288 +0000 UTC m=+34.236974847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:17.800578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:17.800494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:17.800578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:17.800545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:17.801120 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.800622    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:17.801120 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:17.800762    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:18.799903 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:18.799873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:18.800073 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:18.799980    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:19.800638 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:19.800594    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:19.801100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:19.800681    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:19.801100 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:19.800696    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:19.801100 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:19.800791    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:20.801006 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.800825    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:20.801402 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:20.801095    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:20.889750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.888974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lct5p" event={"ID":"f91cc570-5cdc-41a7-8076-28d2370630f7","Type":"ContainerStarted","Data":"3594e6f12ddafad30289309da7fc417e65d0bd3c76e6c3ff6545bd3f86132b58"}
Sep 15 12:02:20.893625 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.893572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"b10cc5a0ab4770150c318bfe16abf5348e8000d01e363720f52de8bab69fb093"}
Sep 15 12:02:20.895109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.894618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"7b0c8b2da1c26c041b8d0334b0eacf3822017231b1e6acff0bcbc404e99baaf1"}
Sep 15 12:02:20.895109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.894679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"1368777be0601535d554a97560ae07f9b0effa1acfad1a57bfebef3795dc64f7"}
Sep 15 12:02:20.900252 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.898269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rlnwn" event={"ID":"20da350e-23cd-4bfa-8e7c-783b11bb435c","Type":"ContainerStarted","Data":"77b9c5244b578a849814073ea441a9700310d4652b805d2f6889fbca5bb6ce9e"}
Sep 15 12:02:20.929867 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.929806    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rlnwn" podStartSLOduration=3.047553705 podStartE2EDuration="20.929790894s" totalImagesPullingTime="17.882237189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:20.929400128 +0000 UTC m=+21.625959728" watchObservedRunningTime="2026-09-15 12:02:20.929790894 +0000 UTC m=+21.626350473"
Sep 15 12:02:20.930137 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:20.930097    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lct5p" podStartSLOduration=2.92117468 podStartE2EDuration="20.930081533s" totalImagesPullingTime="18.008906853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:20.911809627 +0000 UTC m=+21.608369229" watchObservedRunningTime="2026-09-15 12:02:20.930081533 +0000 UTC m=+21.626641114"
Sep 15 12:02:21.800544 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.800507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:21.800716 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.800549    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:21.800716 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:21.800629    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:21.800794 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:21.800746    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:21.902108 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.902071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xw4k7" event={"ID":"144046aa-3d7c-4925-8259-da5480d92d3e","Type":"ContainerStarted","Data":"a43adc4306d775932812c1bb365bf964f8891292740a4d480f974889e978bc00"}
Sep 15 12:02:21.903596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.903570    2583 generic.go:356] "Generic (PLEG): container finished" podID="15c02746-890f-4bfa-9bbe-c20980af3d45" containerID="2c8d27358d0eefae4f174cdd3c62d577b9081fbaed1a87e8347b052f939ef858" exitCode=0
Sep 15 12:02:21.903737 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.903652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerDied","Data":"2c8d27358d0eefae4f174cdd3c62d577b9081fbaed1a87e8347b052f939ef858"}
Sep 15 12:02:21.905183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.905158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wcpvh" event={"ID":"54ab5368-9c3b-4dbe-8740-e89279ab3c6e","Type":"ContainerStarted","Data":"06b76fab3999159f9c0816e89d5fd3fcbb81d50158ee1cc2d2c0b6297413335a"}
Sep 15 12:02:21.908403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.908380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"1cd67051dccb2ff6f5e180d204fed1702214ff72b93daaa017d3b66da2bee1d1"}
Sep 15 12:02:21.908501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.908462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"c9f8731bd9c5df52b843dd1fdf17fb499e91185cd3b0a07e03ce6c7ca7f63c9f"}
Sep 15 12:02:21.908557 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.908512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"ca149f7826980197a6a281eced15562fa74bfb81d5cfca70cd8cfbbdcb3f657b"}
Sep 15 12:02:21.909793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.909768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88" event={"ID":"c57ce7a1-dcfc-4485-8251-2f68887d51ac","Type":"ContainerStarted","Data":"bbd7e0746c1fd23d97a6c8e48fe45fce840902b517fa2525c7743f775eeb9acb"}
Sep 15 12:02:21.911161 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.911142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f79s4" event={"ID":"ea5ba13e-d405-4fb7-9030-811b949df8d8","Type":"ContainerStarted","Data":"9c16e600558080ad6f1d03b091c3655c710f2541db00a1ec0812804f988779ee"}
Sep 15 12:02:21.914638 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.914615    2583 generic.go:356] "Generic (PLEG): container finished" podID="0c79d56c1882a8150a15d2980c27bc5c" containerID="4edcd9e143cea647903cb2b9ea25e9da949ccf2b4bbb7fa3a812b887e1fe5db8" exitCode=0
Sep 15 12:02:21.914730 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.914652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"0c79d56c1882a8150a15d2980c27bc5c","Type":"ContainerDied","Data":"4edcd9e143cea647903cb2b9ea25e9da949ccf2b4bbb7fa3a812b887e1fe5db8"}
Sep 15 12:02:21.916234 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.916214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxmpx" event={"ID":"ca52c69a-01eb-448e-86c0-7a8018d39ad8","Type":"ContainerStarted","Data":"4a9e308820c1557c711d1aaf1f52dbb3b8af77874d69a51cd968c4538f21657b"}
Sep 15 12:02:21.917078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.917017    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xw4k7" podStartSLOduration=4.039317352 podStartE2EDuration="21.917005671s" totalImagesPullingTime="17.877688319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:21.916835257 +0000 UTC m=+22.613394837" watchObservedRunningTime="2026-09-15 12:02:21.917005671 +0000 UTC m=+22.613565248"
Sep 15 12:02:21.946458 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.946408    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f79s4" podStartSLOduration=4.061121424 podStartE2EDuration="21.946389229s" totalImagesPullingTime="17.885267805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:21.946003865 +0000 UTC m=+22.642563458" watchObservedRunningTime="2026-09-15 12:02:21.946389229 +0000 UTC m=+22.642948808"
Sep 15 12:02:21.959787 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:21.959739    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wcpvh" podStartSLOduration=4.083890622 podStartE2EDuration="21.959718534s" totalImagesPullingTime="17.875827912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:21.959220949 +0000 UTC m=+22.655780540" watchObservedRunningTime="2026-09-15 12:02:21.959718534 +0000 UTC m=+22.656278112"
Sep 15 12:02:22.000162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:22.000119    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bxmpx" podStartSLOduration=11.968570934 podStartE2EDuration="22.000103573s" totalImagesPullingTime="10.031532639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:21.999788797 +0000 UTC m=+22.696348376" watchObservedRunningTime="2026-09-15 12:02:22.000103573 +0000 UTC m=+22.696663152"
Sep 15 12:02:22.800739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:22.800706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:22.800912 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:22.800841    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:22.920744 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:22.920699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" event={"ID":"0c79d56c1882a8150a15d2980c27bc5c","Type":"ContainerStarted","Data":"b8717d3d5f017470138d925685bbc45df5e4cda816e69883874ae1331a5a7a4d"}
Sep 15 12:02:22.936350 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:22.936300    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-229.ec2.internal" podStartSLOduration=21.936285658 podStartE2EDuration="21.936285658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:22.935600057 +0000 UTC m=+23.632159636" watchObservedRunningTime="2026-09-15 12:02:22.936285658 +0000 UTC m=+23.632845234"
Sep 15 12:02:23.800259 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:23.800043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:23.800445 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:23.800080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:23.800445 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:23.800344    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:23.800572 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:23.800450    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:23.927109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:23.927059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"751c306391df0a9fd8abbcf62e2fba3fc1186827dcab1b56de920c4f817fd3fa"}
Sep 15 12:02:24.439458 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:24.439422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:24.440080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:24.440052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:24.799936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:24.799846    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:24.800116 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:24.799988    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:24.931048 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:24.930736    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:24.931481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:24.931187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bxmpx"
Sep 15 12:02:25.186703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.186681    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 12:02:25.776287 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.775898    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T12:02:25.186698519Z","UUID":"265d74cc-7cb4-49fa-acb6-f777c119ee19","Handler":null,"Name":"","Endpoint":""}
Sep 15 12:02:25.777449 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.777419    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 12:02:25.777567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.777457    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 12:02:25.800241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.800214    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:25.800363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.800216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:25.800363 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:25.800320    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:25.800463 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:25.800426    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:25.936535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.936493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" event={"ID":"c942e6b8-5d26-4666-8a62-5fdf72686e63","Type":"ContainerStarted","Data":"e33e83e35c5f557adca817089b3106b807948b0bf91f8565f8a92f1113cd896b"}
Sep 15 12:02:25.937074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.936769    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:25.937074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.936834    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:25.939386 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.939280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88" event={"ID":"c57ce7a1-dcfc-4485-8251-2f68887d51ac","Type":"ContainerStarted","Data":"c29bca9ce4f82f4ab16d427bf9ff74ce345d0739602c1b8b7c958b2ebe9da96f"}
Sep 15 12:02:25.953639 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.953613    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:25.954783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.954761    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:25.974130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:25.974088    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" podStartSLOduration=8.045363633 podStartE2EDuration="25.974060988s" totalImagesPullingTime="17.928697355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:25.97392415 +0000 UTC m=+26.670483761" watchObservedRunningTime="2026-09-15 12:02:25.974060988 +0000 UTC m=+26.670620567"
Sep 15 12:02:26.800415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:26.800380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:26.800577 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:26.800521    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:26.941384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:26.941357    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 12:02:27.431517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.431225    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g2n6z"]
Sep 15 12:02:27.431517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.431403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:27.432078 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:27.431521    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:27.433220 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.433193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-66f9b"]
Sep 15 12:02:27.433317 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.433301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:27.433430 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:27.433404    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:27.445952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.445926    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mjfj7"]
Sep 15 12:02:27.446143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.446077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:27.446277 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:27.446167    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:27.944975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.944898    2583 generic.go:356] "Generic (PLEG): container finished" podID="15c02746-890f-4bfa-9bbe-c20980af3d45" containerID="0b93067f94e4fc05d79cedf648fcb30ba260437c8e707376ec9025c5f26e83e8" exitCode=0
Sep 15 12:02:27.945675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.944993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerDied","Data":"0b93067f94e4fc05d79cedf648fcb30ba260437c8e707376ec9025c5f26e83e8"}
Sep 15 12:02:27.945675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:27.945114    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 12:02:28.800913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:28.800879    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:28.800913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:28.800912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:28.801185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:28.800912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:28.801185 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:28.801043    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:28.801185 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:28.801120    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:28.801313 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:28.801211    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:28.949439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:28.949405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88" event={"ID":"c57ce7a1-dcfc-4485-8251-2f68887d51ac","Type":"ContainerStarted","Data":"35d4055bbc8adb6bff1a90e5711f497d5206cb6d8b134a78bf8716f0c4ae6e06"}
Sep 15 12:02:28.970748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:28.970704    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g6c88" podStartSLOduration=4.124063233 podStartE2EDuration="28.97069019s" totalImagesPullingTime="24.846626957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:28.970177848 +0000 UTC m=+29.666737444" watchObservedRunningTime="2026-09-15 12:02:28.97069019 +0000 UTC m=+29.667249769"
Sep 15 12:02:29.953869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:29.953678    2583 generic.go:356] "Generic (PLEG): container finished" podID="15c02746-890f-4bfa-9bbe-c20980af3d45" containerID="9d9a7a0ccb31df57cf507942997e68f6c1d221db05e2f9fad7ab3d93b06b88b7" exitCode=0
Sep 15 12:02:29.954336 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:29.953761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerDied","Data":"9d9a7a0ccb31df57cf507942997e68f6c1d221db05e2f9fad7ab3d93b06b88b7"}
Sep 15 12:02:30.800261 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:30.800228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:30.800261 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:30.800260    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:30.800467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:30.800341    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:30.800467 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:30.800343    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:30.800467 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:30.800426    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:30.800566 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:30.800491    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:31.027099 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:31.027068    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:02:31.027466 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:31.027291    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 12:02:31.046672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:31.046633    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" podUID="c942e6b8-5d26-4666-8a62-5fdf72686e63" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 15 12:02:31.055177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:31.055101    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf" podUID="c942e6b8-5d26-4666-8a62-5fdf72686e63" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 15 12:02:31.960132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:31.960095    2583 generic.go:356] "Generic (PLEG): container finished" podID="15c02746-890f-4bfa-9bbe-c20980af3d45" containerID="33ea73e2e4eadb6d3aa1e16c04a1366223dfc0222c18c267888c17dc2569e7c4" exitCode=0
Sep 15 12:02:31.960307 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:31.960162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerDied","Data":"33ea73e2e4eadb6d3aa1e16c04a1366223dfc0222c18c267888c17dc2569e7c4"}
Sep 15 12:02:32.800876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:32.800845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:32.801267 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:32.800845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:32.801267 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:32.800954    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2n6z" podUID="59f50a12-82fe-48ec-8b9f-26aa7c1133d4"
Sep 15 12:02:32.801267 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:32.800845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:32.801267 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:32.801017    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mjfj7" podUID="4e8e54dd-ad17-46b3-b460-0da6e0d0efef"
Sep 15 12:02:32.801267 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:32.801117    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-66f9b" podUID="cec4d24d-6cbf-485b-9754-3ab98fce5a6f"
Sep 15 12:02:33.454230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.454139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:33.454230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.454189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:33.454433 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.454291    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:33.454433 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.454307    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:33.454433 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.454348    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs podName:59f50a12-82fe-48ec-8b9f-26aa7c1133d4 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.454332439 +0000 UTC m=+66.150892003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs") pod "network-metrics-daemon-g2n6z" (UID: "59f50a12-82fe-48ec-8b9f-26aa7c1133d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:02:33.454433 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.454374    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret podName:cec4d24d-6cbf-485b-9754-3ab98fce5a6f nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.454354565 +0000 UTC m=+66.150914124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret") pod "global-pull-secret-syncer-66f9b" (UID: "cec4d24d-6cbf-485b-9754-3ab98fce5a6f") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:02:33.555554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.555516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:33.555729 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.555708    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:02:33.555773 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.555735    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:02:33.555773 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.555749    2583 projected.go:196] Error preparing data for projected volume kube-api-access-74tkg for pod openshift-network-diagnostics/network-check-target-mjfj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:33.555855 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.555817    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg podName:4e8e54dd-ad17-46b3-b460-0da6e0d0efef nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.555795116 +0000 UTC m=+66.252354674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-74tkg" (UniqueName: "kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg") pod "network-check-target-mjfj7" (UID: "4e8e54dd-ad17-46b3-b460-0da6e0d0efef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:02:33.639603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.639575    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-229.ec2.internal" event="NodeReady"
Sep 15 12:02:33.639785 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.639746    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 12:02:33.675173 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.675135    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"]
Sep 15 12:02:33.699245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.699193    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"]
Sep 15 12:02:33.699898 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.699680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.702651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.702613    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 12:02:33.702651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.702619    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.702961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.702943    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bzf8p\""
Sep 15 12:02:33.703056 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.703000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.703250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.703229    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 12:02:33.715177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.715149    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8495c67bfc-xtngb"]
Sep 15 12:02:33.715649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.715624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.719472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.719446    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 12:02:33.719723 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.719706    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cfppq\""
Sep 15 12:02:33.719939 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.719923    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.720164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.720148    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.720392 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.720356    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 12:02:33.730897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.730874    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-847cffd544-fxqfj"]
Sep 15 12:02:33.731049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.731009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.733341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.733316    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 12:02:33.733483 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.733361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 12:02:33.733483 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.733384    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s8xfw\""
Sep 15 12:02:33.733483 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.733418    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 12:02:33.738457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.738431    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 12:02:33.752036 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.752000    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"]
Sep 15 12:02:33.752140 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.752101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.755365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755342    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 12:02:33.755365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ht528\""
Sep 15 12:02:33.755365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755363    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.755606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755391    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 12:02:33.755606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755391    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 12:02:33.755606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 12:02:33.755755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.755709    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.774973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.774954    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"]
Sep 15 12:02:33.775098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.775004    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"]
Sep 15 12:02:33.775098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.775054    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-s5npn"]
Sep 15 12:02:33.775221 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.775176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.777792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.777773    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.778719 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.778695    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-59pzv\""
Sep 15 12:02:33.778808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.778740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 12:02:33.778808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.778697    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.778808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.778760    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 12:02:33.785586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.785569    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"]
Sep 15 12:02:33.785679 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.785660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.788041 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.788013    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.788157 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.788046    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 12:02:33.788217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.788156    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.788322 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.788305    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-94jnn\""
Sep 15 12:02:33.788821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.788803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 12:02:33.795172 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.795153    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 12:02:33.805892 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.805865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"
Sep 15 12:02:33.808046 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.808015    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jv2wg\""
Sep 15 12:02:33.808196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.808177    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.808715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.808559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.810319 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.810301    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8zw5m"]
Sep 15 12:02:33.830440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.830414    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zcjwh"]
Sep 15 12:02:33.830571 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.830541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:33.832825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.832800    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 12:02:33.832935 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.832802    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.833379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.833364    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.833895 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.833875    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 12:02:33.834040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.834009    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lsp4m\""
Sep 15 12:02:33.841002 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.840982    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 12:02:33.850742 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.850697    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"]
Sep 15 12:02:33.850843 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.850761    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"]
Sep 15 12:02:33.850896 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.850871    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:33.853528 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.853506    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.853808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.853791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 12:02:33.853948 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.853933    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-swgf8\""
Sep 15 12:02:33.854226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.854204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.857515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23e5d6de-da28-4ac9-b31d-544108426598-ca-trust-extracted\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.857635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-trusted-ca\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.857635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-installation-pull-secrets\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.857635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.857808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-stats-auth\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.857808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-image-registry-private-configuration\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.857808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47fdb44b-602b-4b18-b9f7-e1a11db46547-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.857808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jsq5\" (UniqueName: \"kubernetes.io/projected/23b96874-988f-4db1-a224-bb08ea644d09-kube-api-access-6jsq5\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.857808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmm4\" (UniqueName: \"kubernetes.io/projected/ef818076-7989-462f-b9f6-92a7cc8352d8-kube-api-access-rvmm4\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-registry-certificates\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmmz\" (UniqueName: \"kubernetes.io/projected/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-kube-api-access-6nmmz\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23b96874-988f-4db1-a224-bb08ea644d09-serving-cert\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-default-certificate\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.857991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23b96874-988f-4db1-a224-bb08ea644d09-config\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ht7\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-kube-api-access-p5ht7\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.858187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44w2q\" (UniqueName: \"kubernetes.io/projected/47fdb44b-602b-4b18-b9f7-e1a11db46547-kube-api-access-44w2q\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.858609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.858609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.858609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-bound-sa-token\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.858609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.858354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47fdb44b-602b-4b18-b9f7-e1a11db46547-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.875282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.875257    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"]
Sep 15 12:02:33.875440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.875422    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"
Sep 15 12:02:33.875499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.875440    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:02:33.877909 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.877881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.877909 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.877888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 12:02:33.878084 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.877953    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sklqc\""
Sep 15 12:02:33.878084 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.877999    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.898436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.898410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8495c67bfc-xtngb"]
Sep 15 12:02:33.898541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.898445    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-847cffd544-fxqfj"]
Sep 15 12:02:33.898541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.898463    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8ckhc"]
Sep 15 12:02:33.898541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.898508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:33.901107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.901080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lcgbm\""
Sep 15 12:02:33.901206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.901105    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 12:02:33.901206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.901080    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:33.901316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.901090    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 12:02:33.923224 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923203    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"]
Sep 15 12:02:33.923316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923236    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"]
Sep 15 12:02:33.923316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923258    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-s5npn"]
Sep 15 12:02:33.923316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zcjwh"]
Sep 15 12:02:33.923316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923297    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8ckhc"]
Sep 15 12:02:33.923542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923318    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"]
Sep 15 12:02:33.923542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923344    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8zw5m"]
Sep 15 12:02:33.923542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.923403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:33.925477 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.925458    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 12:02:33.925579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.925496    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 12:02:33.925646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.925632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gwv7\""
Sep 15 12:02:33.959674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.959674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.959674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-bound-sa-token\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b404c4b-5d10-407c-9193-4eb1fe994a97-service-ca-bundle\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47fdb44b-602b-4b18-b9f7-e1a11db46547-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-trusted-ca\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-installation-pull-secrets\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.959782    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.959804    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8495c67bfc-xtngb: secret "image-registry-tls" not found
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-stats-auth\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.959870    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls podName:23e5d6de-da28-4ac9-b31d-544108426598 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:34.459849204 +0000 UTC m=+35.156408779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls") pod "image-registry-8495c67bfc-xtngb" (UID: "23e5d6de-da28-4ac9-b31d-544108426598") : secret "image-registry-tls" not found
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqd56\" (UniqueName: \"kubernetes.io/projected/81040b81-30da-48ef-963b-d42e89a16b58-kube-api-access-kqd56\") pod \"ingress-canary-zcjwh\" (UID: \"81040b81-30da-48ef-963b-d42e89a16b58\") " pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:33.959928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jznz8\" (UniqueName: \"kubernetes.io/projected/0b404c4b-5d10-407c-9193-4eb1fe994a97-kube-api-access-jznz8\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.960419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jsq5\" (UniqueName: \"kubernetes.io/projected/23b96874-988f-4db1-a224-bb08ea644d09-kube-api-access-6jsq5\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.960419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.959980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmm4\" (UniqueName: \"kubernetes.io/projected/ef818076-7989-462f-b9f6-92a7cc8352d8-kube-api-access-rvmm4\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.960419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-default-certificate\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.960564 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzshw\" (UniqueName: \"kubernetes.io/projected/0897102c-f1f6-4be1-bd84-68fe8cc4dc3c-kube-api-access-dzshw\") pod \"network-check-source-6f8b768455-jt8jw\" (UID: \"0897102c-f1f6-4be1-bd84-68fe8cc4dc3c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"
Sep 15 12:02:33.960564 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmmz\" (UniqueName: \"kubernetes.io/projected/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-kube-api-access-6nmmz\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.960667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23b96874-988f-4db1-a224-bb08ea644d09-serving-cert\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.960667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0b404c4b-5d10-407c-9193-4eb1fe994a97-snapshots\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.960667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b404c4b-5d10-407c-9193-4eb1fe994a97-serving-cert\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.960667 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.960640    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:33.960857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5570725-5835-48e5-8a10-599f37cffed7-serving-cert\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:33.960857 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.960692    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls podName:da6c3092-9ef2-4a81-9a90-0deecff7c4bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:34.460675842 +0000 UTC m=+35.157235403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b5cnn" (UID: "da6c3092-9ef2-4a81-9a90-0deecff7c4bc") : secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:33.960857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ht7\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-kube-api-access-p5ht7\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.961011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.960886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrmq\" (UniqueName: \"kubernetes.io/projected/c5570725-5835-48e5-8a10-599f37cffed7-kube-api-access-grrmq\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:33.961410 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.961696 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2mr\" (UniqueName: \"kubernetes.io/projected/40dc7c0c-4616-4f6e-bde1-568ad50936af-kube-api-access-df2mr\") pod \"volume-data-source-validator-86695b7cc7-x664q\" (UID: \"40dc7c0c-4616-4f6e-bde1-568ad50936af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"
Sep 15 12:02:33.961779 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23e5d6de-da28-4ac9-b31d-544108426598-ca-trust-extracted\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.961779 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-image-registry-private-configuration\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.961779 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47fdb44b-602b-4b18-b9f7-e1a11db46547-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.961936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b404c4b-5d10-407c-9193-4eb1fe994a97-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.961936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-registry-certificates\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.961936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.961936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44w2q\" (UniqueName: \"kubernetes.io/projected/47fdb44b-602b-4b18-b9f7-e1a11db46547-kube-api-access-44w2q\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.961936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81040b81-30da-48ef-963b-d42e89a16b58-cert\") pod \"ingress-canary-zcjwh\" (UID: \"81040b81-30da-48ef-963b-d42e89a16b58\") " pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:33.962185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-trusted-ca\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.962185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5570725-5835-48e5-8a10-599f37cffed7-config\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:33.962185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.961996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b404c4b-5d10-407c-9193-4eb1fe994a97-tmp\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:33.962185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.962041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5570725-5835-48e5-8a10-599f37cffed7-trusted-ca\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:33.962185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.962080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23b96874-988f-4db1-a224-bb08ea644d09-config\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.962185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.962110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.962470 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.962221    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 12:02:33.962470 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.962266    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:34.462251874 +0000 UTC m=+35.158811430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : secret "router-metrics-certs-default" not found
Sep 15 12:02:33.962470 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.962356    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23e5d6de-da28-4ac9-b31d-544108426598-ca-trust-extracted\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.963338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.962838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/23b96874-988f-4db1-a224-bb08ea644d09-config\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.963338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.963116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-registry-certificates\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.963338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.963142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47fdb44b-602b-4b18-b9f7-e1a11db46547-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.963338 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:33.963265    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:34.463249651 +0000 UTC m=+35.159809226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : configmap references non-existent config key: service-ca.crt
Sep 15 12:02:33.964919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.964891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23b96874-988f-4db1-a224-bb08ea644d09-serving-cert\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.965197 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.965176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-stats-auth\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.965374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.965349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-default-certificate\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.965579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.965539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47fdb44b-602b-4b18-b9f7-e1a11db46547-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:33.966297 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.966270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-installation-pull-secrets\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.966413 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.966337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-image-registry-private-configuration\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.969049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.969002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-bound-sa-token\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.969211 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.969190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jsq5\" (UniqueName: \"kubernetes.io/projected/23b96874-988f-4db1-a224-bb08ea644d09-kube-api-access-6jsq5\") pod \"service-ca-operator-cf7c776b7-f9g8r\" (UID: \"23b96874-988f-4db1-a224-bb08ea644d09\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:33.969672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.969495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ht7\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-kube-api-access-p5ht7\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:33.969672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.969573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmmz\" (UniqueName: \"kubernetes.io/projected/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-kube-api-access-6nmmz\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:33.970441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.970420    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmm4\" (UniqueName: \"kubernetes.io/projected/ef818076-7989-462f-b9f6-92a7cc8352d8-kube-api-access-rvmm4\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:33.970535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:33.970424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44w2q\" (UniqueName: \"kubernetes.io/projected/47fdb44b-602b-4b18-b9f7-e1a11db46547-kube-api-access-44w2q\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ml4vj\" (UID: \"47fdb44b-602b-4b18-b9f7-e1a11db46547\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:34.011586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.011556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"
Sep 15 12:02:34.028724 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.028696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b813070-33ae-426d-ad4a-1fba43dfc263-config-volume\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b404c4b-5d10-407c-9193-4eb1fe994a97-service-ca-bundle\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqd56\" (UniqueName: \"kubernetes.io/projected/81040b81-30da-48ef-963b-d42e89a16b58-kube-api-access-kqd56\") pod \"ingress-canary-zcjwh\" (UID: \"81040b81-30da-48ef-963b-d42e89a16b58\") " pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jznz8\" (UniqueName: \"kubernetes.io/projected/0b404c4b-5d10-407c-9193-4eb1fe994a97-kube-api-access-jznz8\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b813070-33ae-426d-ad4a-1fba43dfc263-tmp-dir\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzshw\" (UniqueName: \"kubernetes.io/projected/0897102c-f1f6-4be1-bd84-68fe8cc4dc3c-kube-api-access-dzshw\") pod \"network-check-source-6f8b768455-jt8jw\" (UID: \"0897102c-f1f6-4be1-bd84-68fe8cc4dc3c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0b404c4b-5d10-407c-9193-4eb1fe994a97-snapshots\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b404c4b-5d10-407c-9193-4eb1fe994a97-serving-cert\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4hw\" (UniqueName: \"kubernetes.io/projected/3b813070-33ae-426d-ad4a-1fba43dfc263-kube-api-access-6w4hw\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd6gg\" (UniqueName: \"kubernetes.io/projected/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-kube-api-access-kd6gg\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5570725-5835-48e5-8a10-599f37cffed7-serving-cert\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrmq\" (UniqueName: \"kubernetes.io/projected/c5570725-5835-48e5-8a10-599f37cffed7-kube-api-access-grrmq\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2mr\" (UniqueName: \"kubernetes.io/projected/40dc7c0c-4616-4f6e-bde1-568ad50936af-kube-api-access-df2mr\") pod \"volume-data-source-validator-86695b7cc7-x664q\" (UID: \"40dc7c0c-4616-4f6e-bde1-568ad50936af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"
Sep 15 12:02:34.067633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b404c4b-5d10-407c-9193-4eb1fe994a97-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81040b81-30da-48ef-963b-d42e89a16b58-cert\") pod \"ingress-canary-zcjwh\" (UID: \"81040b81-30da-48ef-963b-d42e89a16b58\") " pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5570725-5835-48e5-8a10-599f37cffed7-config\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b404c4b-5d10-407c-9193-4eb1fe994a97-tmp\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.063967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5570725-5835-48e5-8a10-599f37cffed7-trusted-ca\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.064777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0b404c4b-5d10-407c-9193-4eb1fe994a97-snapshots\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.065076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b404c4b-5d10-407c-9193-4eb1fe994a97-service-ca-bundle\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.067363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b404c4b-5d10-407c-9193-4eb1fe994a97-tmp\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.068542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.067958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c5570725-5835-48e5-8a10-599f37cffed7-config\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.069592 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.069351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b404c4b-5d10-407c-9193-4eb1fe994a97-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.070715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.070687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5570725-5835-48e5-8a10-599f37cffed7-trusted-ca\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.071068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.070864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b404c4b-5d10-407c-9193-4eb1fe994a97-serving-cert\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.071068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.070998    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81040b81-30da-48ef-963b-d42e89a16b58-cert\") pod \"ingress-canary-zcjwh\" (UID: \"81040b81-30da-48ef-963b-d42e89a16b58\") " pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:34.071759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.071704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c5570725-5835-48e5-8a10-599f37cffed7-serving-cert\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.073973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.073949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jznz8\" (UniqueName: \"kubernetes.io/projected/0b404c4b-5d10-407c-9193-4eb1fe994a97-kube-api-access-jznz8\") pod \"insights-operator-5844b856db-s5npn\" (UID: \"0b404c4b-5d10-407c-9193-4eb1fe994a97\") " pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.074565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.074499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqd56\" (UniqueName: \"kubernetes.io/projected/81040b81-30da-48ef-963b-d42e89a16b58-kube-api-access-kqd56\") pod \"ingress-canary-zcjwh\" (UID: \"81040b81-30da-48ef-963b-d42e89a16b58\") " pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:34.074799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.074778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzshw\" (UniqueName: \"kubernetes.io/projected/0897102c-f1f6-4be1-bd84-68fe8cc4dc3c-kube-api-access-dzshw\") pod \"network-check-source-6f8b768455-jt8jw\" (UID: \"0897102c-f1f6-4be1-bd84-68fe8cc4dc3c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"
Sep 15 12:02:34.075285 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.075265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2mr\" (UniqueName: \"kubernetes.io/projected/40dc7c0c-4616-4f6e-bde1-568ad50936af-kube-api-access-df2mr\") pod \"volume-data-source-validator-86695b7cc7-x664q\" (UID: \"40dc7c0c-4616-4f6e-bde1-568ad50936af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"
Sep 15 12:02:34.077730 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.077711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrmq\" (UniqueName: \"kubernetes.io/projected/c5570725-5835-48e5-8a10-599f37cffed7-kube-api-access-grrmq\") pod \"console-operator-696884fb9c-8zw5m\" (UID: \"c5570725-5835-48e5-8a10-599f37cffed7\") " pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.095819 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.095784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-s5npn"
Sep 15 12:02:34.116934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.116904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"
Sep 15 12:02:34.142150 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.141253    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:34.161896 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.161779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zcjwh"
Sep 15 12:02:34.165406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:34.165522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b813070-33ae-426d-ad4a-1fba43dfc263-tmp-dir\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.165522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4hw\" (UniqueName: \"kubernetes.io/projected/3b813070-33ae-426d-ad4a-1fba43dfc263-kube-api-access-6w4hw\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.165522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd6gg\" (UniqueName: \"kubernetes.io/projected/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-kube-api-access-kd6gg\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:34.165679 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.165545    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 12:02:34.165679 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.165622    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls podName:9436cd56-d74c-4acf-b6bb-4ce6093ba6e1 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:34.665599688 +0000 UTC m=+35.362159249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xcxvn" (UID: "9436cd56-d74c-4acf-b6bb-4ce6093ba6e1") : secret "samples-operator-tls" not found
Sep 15 12:02:34.165780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.165780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b813070-33ae-426d-ad4a-1fba43dfc263-config-volume\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.165876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.165741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b813070-33ae-426d-ad4a-1fba43dfc263-tmp-dir\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.165876 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.165820    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:02:34.165876 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.165864    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls podName:3b813070-33ae-426d-ad4a-1fba43dfc263 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:34.66584981 +0000 UTC m=+35.362409381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls") pod "dns-default-8ckhc" (UID: "3b813070-33ae-426d-ad4a-1fba43dfc263") : secret "dns-default-metrics-tls" not found
Sep 15 12:02:34.166323 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.166302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b813070-33ae-426d-ad4a-1fba43dfc263-config-volume\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.174579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.174551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4hw\" (UniqueName: \"kubernetes.io/projected/3b813070-33ae-426d-ad4a-1fba43dfc263-kube-api-access-6w4hw\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.174733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.174551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd6gg\" (UniqueName: \"kubernetes.io/projected/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-kube-api-access-kd6gg\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:34.185970 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.185945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"
Sep 15 12:02:34.352743 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.352710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r"]
Sep 15 12:02:34.355040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.355007    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj"]
Sep 15 12:02:34.356314 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.356287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b96874_988f_4db1_a224_bb08ea644d09.slice/crio-10a21bd0214c5a96e373b546a960f56ec9674b3d36445576c2358d1436ee02f7 WatchSource:0}: Error finding container 10a21bd0214c5a96e373b546a960f56ec9674b3d36445576c2358d1436ee02f7: Status 404 returned error can't find the container with id 10a21bd0214c5a96e373b546a960f56ec9674b3d36445576c2358d1436ee02f7
Sep 15 12:02:34.357463 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.357440    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fdb44b_602b_4b18_b9f7_e1a11db46547.slice/crio-7e611a4fa35683e2f8ce7e8db15600e2a15be0309c16882e2f51a58ec9927cce WatchSource:0}: Error finding container 7e611a4fa35683e2f8ce7e8db15600e2a15be0309c16882e2f51a58ec9927cce: Status 404 returned error can't find the container with id 7e611a4fa35683e2f8ce7e8db15600e2a15be0309c16882e2f51a58ec9927cce
Sep 15 12:02:34.410955 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.410925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-s5npn"]
Sep 15 12:02:34.413655 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.413627    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b404c4b_5d10_407c_9193_4eb1fe994a97.slice/crio-b85d936f7928aa49f12d66af2eef07a390cb0e5ae327e781fc1d1ccbb9724d1f WatchSource:0}: Error finding container b85d936f7928aa49f12d66af2eef07a390cb0e5ae327e781fc1d1ccbb9724d1f: Status 404 returned error can't find the container with id b85d936f7928aa49f12d66af2eef07a390cb0e5ae327e781fc1d1ccbb9724d1f
Sep 15 12:02:34.441072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.441041    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"]
Sep 15 12:02:34.451163 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.451137    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dc7c0c_4616_4f6e_bde1_568ad50936af.slice/crio-740ea9b57c07ea63c13e69c25aa225c9410eac6f2f19b5b92c902a3f5209a2fa WatchSource:0}: Error finding container 740ea9b57c07ea63c13e69c25aa225c9410eac6f2f19b5b92c902a3f5209a2fa: Status 404 returned error can't find the container with id 740ea9b57c07ea63c13e69c25aa225c9410eac6f2f19b5b92c902a3f5209a2fa
Sep 15 12:02:34.468356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.468281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:34.468356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.468340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:34.468541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.468366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:34.468541 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468460    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 12:02:34.468541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.468496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:34.468541 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468534    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.468518773 +0000 UTC m=+36.165078329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : secret "router-metrics-certs-default" not found
Sep 15 12:02:34.468541 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468535    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:34.468732 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468464    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:02:34.468732 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468574    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8495c67bfc-xtngb: secret "image-registry-tls" not found
Sep 15 12:02:34.468732 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468599    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls podName:da6c3092-9ef2-4a81-9a90-0deecff7c4bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.468582057 +0000 UTC m=+36.165141636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b5cnn" (UID: "da6c3092-9ef2-4a81-9a90-0deecff7c4bc") : secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:34.468732 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468618    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.46860863 +0000 UTC m=+36.165168188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : configmap references non-existent config key: service-ca.crt
Sep 15 12:02:34.468732 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.468640    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls podName:23e5d6de-da28-4ac9-b31d-544108426598 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.468630097 +0000 UTC m=+36.165189657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls") pod "image-registry-8495c67bfc-xtngb" (UID: "23e5d6de-da28-4ac9-b31d-544108426598") : secret "image-registry-tls" not found
Sep 15 12:02:34.473644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.473615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q"]
Sep 15 12:02:34.473768 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.473663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"]
Sep 15 12:02:34.473768 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.473749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:34.476139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.476113    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 12:02:34.476262 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.476161    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-45459\""
Sep 15 12:02:34.476414 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.476398    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 12:02:34.479437 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.479416    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8zw5m"]
Sep 15 12:02:34.483517 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.483492    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5570725_5835_48e5_8a10_599f37cffed7.slice/crio-55abe75588f4453e31f534f009d4a5da1e73f9ba26c4d41eae3899bdb289047f WatchSource:0}: Error finding container 55abe75588f4453e31f534f009d4a5da1e73f9ba26c4d41eae3899bdb289047f: Status 404 returned error can't find the container with id 55abe75588f4453e31f534f009d4a5da1e73f9ba26c4d41eae3899bdb289047f
Sep 15 12:02:34.507492 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.507377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zcjwh"]
Sep 15 12:02:34.510040 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.510003    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81040b81_30da_48ef_963b_d42e89a16b58.slice/crio-831d97e631251d76799e6051c7404d89c1df0983fa2e538cbaf9304c6d216c50 WatchSource:0}: Error finding container 831d97e631251d76799e6051c7404d89c1df0983fa2e538cbaf9304c6d216c50: Status 404 returned error can't find the container with id 831d97e631251d76799e6051c7404d89c1df0983fa2e538cbaf9304c6d216c50
Sep 15 12:02:34.519574 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.519552    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw"]
Sep 15 12:02:34.522651 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:34.522619    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0897102c_f1f6_4be1_bd84_68fe8cc4dc3c.slice/crio-5ab6927066928fd06dbbf388eac73b206781f807456c5f025916145e7f85de77 WatchSource:0}: Error finding container 5ab6927066928fd06dbbf388eac73b206781f807456c5f025916145e7f85de77: Status 404 returned error can't find the container with id 5ab6927066928fd06dbbf388eac73b206781f807456c5f025916145e7f85de77
Sep 15 12:02:34.526957 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.526930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:34.569781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.569739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:34.569925 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.569792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/31a288ab-9731-41ec-8b6d-cc37c790fe6f-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:34.621895 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.621793    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:34.670978 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.670941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:34.671150 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.671014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:34.671150 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.671064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:34.671150 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.671099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/31a288ab-9731-41ec-8b6d-cc37c790fe6f-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:34.671150 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.671114    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:02:34.671351 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.671172    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls podName:3b813070-33ae-426d-ad4a-1fba43dfc263 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.67115734 +0000 UTC m=+36.367716895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls") pod "dns-default-8ckhc" (UID: "3b813070-33ae-426d-ad4a-1fba43dfc263") : secret "dns-default-metrics-tls" not found
Sep 15 12:02:34.671351 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.671115    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 12:02:34.671351 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.671230    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls podName:9436cd56-d74c-4acf-b6bb-4ce6093ba6e1 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.671218464 +0000 UTC m=+36.367778019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xcxvn" (UID: "9436cd56-d74c-4acf-b6bb-4ce6093ba6e1") : secret "samples-operator-tls" not found
Sep 15 12:02:34.671351 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.671184    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 12:02:34.671351 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:34.671268    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert podName:31a288ab-9731-41ec-8b6d-cc37c790fe6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:35.171258282 +0000 UTC m=+35.867817841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-lfzm7" (UID: "31a288ab-9731-41ec-8b6d-cc37c790fe6f") : secret "networking-console-plugin-cert" not found
Sep 15 12:02:34.671862 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.671839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/31a288ab-9731-41ec-8b6d-cc37c790fe6f-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:34.721779 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.721690    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:34.800850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.800819    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:02:34.801045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.800821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:02:34.803201 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.803180    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 12:02:34.803321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.803307    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l82m\""
Sep 15 12:02:34.803387 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.803374    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k48ps\""
Sep 15 12:02:34.823726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.823696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:34.883606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.883572    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:34.969676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.969637    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" event={"ID":"c5570725-5835-48e5-8a10-599f37cffed7","Type":"ContainerStarted","Data":"55abe75588f4453e31f534f009d4a5da1e73f9ba26c4d41eae3899bdb289047f"}
Sep 15 12:02:34.971104 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.971079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zcjwh" event={"ID":"81040b81-30da-48ef-963b-d42e89a16b58","Type":"ContainerStarted","Data":"831d97e631251d76799e6051c7404d89c1df0983fa2e538cbaf9304c6d216c50"}
Sep 15 12:02:34.972369 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.972308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s5npn" event={"ID":"0b404c4b-5d10-407c-9193-4eb1fe994a97","Type":"ContainerStarted","Data":"b85d936f7928aa49f12d66af2eef07a390cb0e5ae327e781fc1d1ccbb9724d1f"}
Sep 15 12:02:34.973674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.973639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj" event={"ID":"47fdb44b-602b-4b18-b9f7-e1a11db46547","Type":"ContainerStarted","Data":"7e611a4fa35683e2f8ce7e8db15600e2a15be0309c16882e2f51a58ec9927cce"}
Sep 15 12:02:34.975005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.974983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q" event={"ID":"40dc7c0c-4616-4f6e-bde1-568ad50936af","Type":"ContainerStarted","Data":"740ea9b57c07ea63c13e69c25aa225c9410eac6f2f19b5b92c902a3f5209a2fa"}
Sep 15 12:02:34.976841 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.976812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r" event={"ID":"23b96874-988f-4db1-a224-bb08ea644d09","Type":"ContainerStarted","Data":"10a21bd0214c5a96e373b546a960f56ec9674b3d36445576c2358d1436ee02f7"}
Sep 15 12:02:34.978195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:34.978174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw" event={"ID":"0897102c-f1f6-4be1-bd84-68fe8cc4dc3c","Type":"ContainerStarted","Data":"5ab6927066928fd06dbbf388eac73b206781f807456c5f025916145e7f85de77"}
Sep 15 12:02:35.175969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.175930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:35.176169 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.176123    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 12:02:35.176258 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.176204    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert podName:31a288ab-9731-41ec-8b6d-cc37c790fe6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:36.17618366 +0000 UTC m=+36.872743222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-lfzm7" (UID: "31a288ab-9731-41ec-8b6d-cc37c790fe6f") : secret "networking-console-plugin-cert" not found
Sep 15 12:02:35.480041 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.479991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:35.480218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.480086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:35.480218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.480126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:35.480218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.480155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:35.480411 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.480395    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:35.480478 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.480466    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls podName:da6c3092-9ef2-4a81-9a90-0deecff7c4bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:37.480446313 +0000 UTC m=+38.177005874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b5cnn" (UID: "da6c3092-9ef2-4a81-9a90-0deecff7c4bc") : secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:35.480883 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.480863    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:37.480847657 +0000 UTC m=+38.177407216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : configmap references non-existent config key: service-ca.crt
Sep 15 12:02:35.480973 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.480944    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 12:02:35.481062 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.480976    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:37.480965325 +0000 UTC m=+38.177524885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : secret "router-metrics-certs-default" not found
Sep 15 12:02:35.481062 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.481059    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:02:35.481186 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.481071    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8495c67bfc-xtngb: secret "image-registry-tls" not found
Sep 15 12:02:35.481186 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.481102    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls podName:23e5d6de-da28-4ac9-b31d-544108426598 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:37.481091476 +0000 UTC m=+38.177651037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls") pod "image-registry-8495c67bfc-xtngb" (UID: "23e5d6de-da28-4ac9-b31d-544108426598") : secret "image-registry-tls" not found
Sep 15 12:02:35.682549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.681663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:35.682549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:35.681750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:35.682549 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.681979    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 12:02:35.682549 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.682055    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls podName:9436cd56-d74c-4acf-b6bb-4ce6093ba6e1 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:37.682035941 +0000 UTC m=+38.378595510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xcxvn" (UID: "9436cd56-d74c-4acf-b6bb-4ce6093ba6e1") : secret "samples-operator-tls" not found
Sep 15 12:02:35.682549 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.682464    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:02:35.682549 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:35.682516    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls podName:3b813070-33ae-426d-ad4a-1fba43dfc263 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:37.682498599 +0000 UTC m=+38.379058160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls") pod "dns-default-8ckhc" (UID: "3b813070-33ae-426d-ad4a-1fba43dfc263") : secret "dns-default-metrics-tls" not found
Sep 15 12:02:36.187589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:36.187003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:36.187589 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:36.187173    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 12:02:36.187589 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:36.187245    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert podName:31a288ab-9731-41ec-8b6d-cc37c790fe6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:38.187220601 +0000 UTC m=+38.883780161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-lfzm7" (UID: "31a288ab-9731-41ec-8b6d-cc37c790fe6f") : secret "networking-console-plugin-cert" not found
Sep 15 12:02:37.141642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.141578    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:37.238906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.238849    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:37.499891 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.499789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:37.499891 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.499848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:37.499891 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.499873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:37.500164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.499967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:37.500164 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.499976    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 12:02:37.500164 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500106    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.500087581 +0000 UTC m=+42.196647137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : secret "router-metrics-certs-default" not found
Sep 15 12:02:37.500164 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500128    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.500119658 +0000 UTC m=+42.196679251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : configmap references non-existent config key: service-ca.crt
Sep 15 12:02:37.500409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500161    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:02:37.500409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500172    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:37.500409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500183    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8495c67bfc-xtngb: secret "image-registry-tls" not found
Sep 15 12:02:37.500409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500231    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls podName:da6c3092-9ef2-4a81-9a90-0deecff7c4bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.500214858 +0000 UTC m=+42.196774430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b5cnn" (UID: "da6c3092-9ef2-4a81-9a90-0deecff7c4bc") : secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:37.500409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.500249    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls podName:23e5d6de-da28-4ac9-b31d-544108426598 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.50023961 +0000 UTC m=+42.196799171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls") pod "image-registry-8495c67bfc-xtngb" (UID: "23e5d6de-da28-4ac9-b31d-544108426598") : secret "image-registry-tls" not found
Sep 15 12:02:37.701332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.701292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:37.701513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:37.701385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:37.701513 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.701434    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:02:37.701634 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.701519    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls podName:3b813070-33ae-426d-ad4a-1fba43dfc263 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.701497081 +0000 UTC m=+42.398056643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls") pod "dns-default-8ckhc" (UID: "3b813070-33ae-426d-ad4a-1fba43dfc263") : secret "dns-default-metrics-tls" not found
Sep 15 12:02:37.701634 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.701544    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 12:02:37.701634 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:37.701615    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls podName:9436cd56-d74c-4acf-b6bb-4ce6093ba6e1 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:41.701595356 +0000 UTC m=+42.398154915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xcxvn" (UID: "9436cd56-d74c-4acf-b6bb-4ce6093ba6e1") : secret "samples-operator-tls" not found
Sep 15 12:02:38.205868 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:38.205824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:38.206089 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:38.205999    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 12:02:38.206153 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:38.206100    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert podName:31a288ab-9731-41ec-8b6d-cc37c790fe6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:42.206078497 +0000 UTC m=+42.902638054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-lfzm7" (UID: "31a288ab-9731-41ec-8b6d-cc37c790fe6f") : secret "networking-console-plugin-cert" not found
Sep 15 12:02:38.793529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:38.793488    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:38.854896 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:38.854775    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:41.536676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:41.536639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:41.536689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.536785    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:41.536813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.536837    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:49.536812397 +0000 UTC m=+50.233371967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : configmap references non-existent config key: service-ca.crt
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.536869    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.536878    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8495c67bfc-xtngb: secret "image-registry-tls" not found
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.536906    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:49.536891943 +0000 UTC m=+50.233451498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : secret "router-metrics-certs-default" not found
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:41.536936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.536970    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.537000    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls podName:da6c3092-9ef2-4a81-9a90-0deecff7c4bc nodeName:}" failed. No retries permitted until 2026-09-15 12:02:49.536990789 +0000 UTC m=+50.233550348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b5cnn" (UID: "da6c3092-9ef2-4a81-9a90-0deecff7c4bc") : secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:41.537253 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.537016    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls podName:23e5d6de-da28-4ac9-b31d-544108426598 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:49.537010495 +0000 UTC m=+50.233570055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls") pod "image-registry-8495c67bfc-xtngb" (UID: "23e5d6de-da28-4ac9-b31d-544108426598") : secret "image-registry-tls" not found
Sep 15 12:02:41.738974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:41.738935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:41.739153 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:41.739041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:41.739153 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.739113    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:02:41.739279 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.739176    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 12:02:41.739279 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.739191    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls podName:3b813070-33ae-426d-ad4a-1fba43dfc263 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:49.739171083 +0000 UTC m=+50.435730638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls") pod "dns-default-8ckhc" (UID: "3b813070-33ae-426d-ad4a-1fba43dfc263") : secret "dns-default-metrics-tls" not found
Sep 15 12:02:41.739279 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:41.739238    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls podName:9436cd56-d74c-4acf-b6bb-4ce6093ba6e1 nodeName:}" failed. No retries permitted until 2026-09-15 12:02:49.739219726 +0000 UTC m=+50.435779285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xcxvn" (UID: "9436cd56-d74c-4acf-b6bb-4ce6093ba6e1") : secret "samples-operator-tls" not found
Sep 15 12:02:42.243527 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:42.243487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:42.243674 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:42.243624    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 12:02:42.243713 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:42.243692    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert podName:31a288ab-9731-41ec-8b6d-cc37c790fe6f nodeName:}" failed. No retries permitted until 2026-09-15 12:02:50.243675986 +0000 UTC m=+50.940235547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-lfzm7" (UID: "31a288ab-9731-41ec-8b6d-cc37c790fe6f") : secret "networking-console-plugin-cert" not found
Sep 15 12:02:44.008720 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.008554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" event={"ID":"c5570725-5835-48e5-8a10-599f37cffed7","Type":"ContainerStarted","Data":"63ea1fcc22a19534e9c54564b9f11c3b2cda9c75a224d291b979a9fb1f9c05d1"}
Sep 15 12:02:44.009886 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.009354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:44.010665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.010552    2583 patch_prober.go:28] interesting pod/console-operator-696884fb9c-8zw5m container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Sep 15 12:02:44.010665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.010600    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podUID="c5570725-5835-48e5-8a10-599f37cffed7" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Sep 15 12:02:44.014531 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.012589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zcjwh" event={"ID":"81040b81-30da-48ef-963b-d42e89a16b58","Type":"ContainerStarted","Data":"dae156e924f03432dbe0db0cf6c11278998e33a65341c38f4951166fca0aa564"}
Sep 15 12:02:44.016460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.016441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s5npn" event={"ID":"0b404c4b-5d10-407c-9193-4eb1fe994a97","Type":"ContainerStarted","Data":"d148b9d12f66905ae64a45b27622e4128c67b93a5a8b965df5060036cc04a23a"}
Sep 15 12:02:44.028739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.028259    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podStartSLOduration=9.750263527 podStartE2EDuration="19.028238685s" totalImagesPullingTime="9.277975158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:44.02611175 +0000 UTC m=+44.722671344" watchObservedRunningTime="2026-09-15 12:02:44.028238685 +0000 UTC m=+44.724798263"
Sep 15 12:02:44.067829 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.067775    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zcjwh" podStartSLOduration=1.8166610429999999 podStartE2EDuration="11.067752704s" totalImagesPullingTime="9.251091661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:44.042826902 +0000 UTC m=+44.739386481" watchObservedRunningTime="2026-09-15 12:02:44.067752704 +0000 UTC m=+44.764312283"
Sep 15 12:02:44.142799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.142689    2583 patch_prober.go:28] interesting pod/console-operator-696884fb9c-8zw5m container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body=
Sep 15 12:02:44.142799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:44.142739    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podUID="c5570725-5835-48e5-8a10-599f37cffed7" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": dial tcp 10.132.0.13:8443: connect: connection refused"
Sep 15 12:02:45.021177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.021134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw" event={"ID":"0897102c-f1f6-4be1-bd84-68fe8cc4dc3c","Type":"ContainerStarted","Data":"49187799222ab3d6a2e4f55b009b95de330c4dee774516d5ab0f16581ff24d02"}
Sep 15 12:02:45.030152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.030124    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/0.log"
Sep 15 12:02:45.030324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.030171    2583 generic.go:356] "Generic (PLEG): container finished" podID="c5570725-5835-48e5-8a10-599f37cffed7" containerID="63ea1fcc22a19534e9c54564b9f11c3b2cda9c75a224d291b979a9fb1f9c05d1" exitCode=255
Sep 15 12:02:45.030324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.030259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" event={"ID":"c5570725-5835-48e5-8a10-599f37cffed7","Type":"ContainerDied","Data":"63ea1fcc22a19534e9c54564b9f11c3b2cda9c75a224d291b979a9fb1f9c05d1"}
Sep 15 12:02:45.030696 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.030678    2583 scope.go:117] "RemoveContainer" containerID="63ea1fcc22a19534e9c54564b9f11c3b2cda9c75a224d291b979a9fb1f9c05d1"
Sep 15 12:02:45.032003 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.031981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj" event={"ID":"47fdb44b-602b-4b18-b9f7-e1a11db46547","Type":"ContainerStarted","Data":"2507d1d4a8dd64bf7099ae01d146b9ab1437378dcba731cade7dbec57ef092f8"}
Sep 15 12:02:45.034269 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.034246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q" event={"ID":"40dc7c0c-4616-4f6e-bde1-568ad50936af","Type":"ContainerStarted","Data":"d94e4d134c58ae100cbf0a35aa68923a567e087ddd79d0167cdd388cec590850"}
Sep 15 12:02:45.035930 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.035907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r" event={"ID":"23b96874-988f-4db1-a224-bb08ea644d09","Type":"ContainerStarted","Data":"2dd05d9c71424dabb210d9270c71c28750fd5717627fff2f595b8c3b64f610ca"}
Sep 15 12:02:45.037951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.037896    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-s5npn" podStartSLOduration=10.689644192 podStartE2EDuration="20.03787805s" totalImagesPullingTime="9.348233858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:44.067634685 +0000 UTC m=+44.764194264" watchObservedRunningTime="2026-09-15 12:02:45.03787805 +0000 UTC m=+45.734437610"
Sep 15 12:02:45.040461 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.039017    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jt8jw" podStartSLOduration=10.800108631 podStartE2EDuration="20.039001914s" totalImagesPullingTime="9.238893283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:45.03825897 +0000 UTC m=+45.734818573" watchObservedRunningTime="2026-09-15 12:02:45.039001914 +0000 UTC m=+45.735561492"
Sep 15 12:02:45.040461 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.040119    2583 generic.go:356] "Generic (PLEG): container finished" podID="15c02746-890f-4bfa-9bbe-c20980af3d45" containerID="3de37937da1c88fe3716dcac84ef5d8af5d1576870dfe007e5e85995bf00870b" exitCode=0
Sep 15 12:02:45.040461 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.040180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerDied","Data":"3de37937da1c88fe3716dcac84ef5d8af5d1576870dfe007e5e85995bf00870b"}
Sep 15 12:02:45.054836 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.054789    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-x664q" podStartSLOduration=10.744437008 podStartE2EDuration="20.054759234s" totalImagesPullingTime="9.310322226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:45.053312089 +0000 UTC m=+45.749871678" watchObservedRunningTime="2026-09-15 12:02:45.054759234 +0000 UTC m=+45.751318810"
Sep 15 12:02:45.072219 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.072139    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj" podStartSLOduration=10.636759112 podStartE2EDuration="20.072119957s" totalImagesPullingTime="9.435360845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:45.070653172 +0000 UTC m=+45.767212753" watchObservedRunningTime="2026-09-15 12:02:45.072119957 +0000 UTC m=+45.768679536"
Sep 15 12:02:45.092939 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.092658    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r" podStartSLOduration=10.663148522 podStartE2EDuration="20.092636255s" totalImagesPullingTime="9.429487733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:45.091502654 +0000 UTC m=+45.788062262" watchObservedRunningTime="2026-09-15 12:02:45.092636255 +0000 UTC m=+45.789195834"
Sep 15 12:02:45.140485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.140411    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:45.196773 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.196741    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:45.277203 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.277157    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4w7s9"]
Sep 15 12:02:45.304539 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.304507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.305183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.304397    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4w7s9"]
Sep 15 12:02:45.307312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.307050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 12:02:45.307312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.307101    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 12:02:45.307312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.307101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcl7\""
Sep 15 12:02:45.307545 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.307529    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:45.348223 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.348145    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"]
Sep 15 12:02:45.360275 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.360252    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"]
Sep 15 12:02:45.360403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.360347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"
Sep 15 12:02:45.362699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.362667    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 12:02:45.362826 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.362809    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gng9q\""
Sep 15 12:02:45.362995 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.362974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 12:02:45.377549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.377524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmg49\" (UniqueName: \"kubernetes.io/projected/eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5-kube-api-access-nmg49\") pod \"migrator-8b4d697c5-ph8vj\" (UID: \"eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"
Sep 15 12:02:45.377661 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.377599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef233600-bc63-4e81-9078-b87df9e76dbf-crio-socket\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.377714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.377678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.377764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.377719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef233600-bc63-4e81-9078-b87df9e76dbf-data-volume\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.377814 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.377764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.377884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.377871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m7kh\" (UniqueName: \"kubernetes.io/projected/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-api-access-5m7kh\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.404785 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.404743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:45.478256 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m7kh\" (UniqueName: \"kubernetes.io/projected/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-api-access-5m7kh\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.478436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmg49\" (UniqueName: \"kubernetes.io/projected/eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5-kube-api-access-nmg49\") pod \"migrator-8b4d697c5-ph8vj\" (UID: \"eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"
Sep 15 12:02:45.478549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef233600-bc63-4e81-9078-b87df9e76dbf-crio-socket\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.478602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.478654 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef233600-bc63-4e81-9078-b87df9e76dbf-data-volume\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.478707 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.478754 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.478728    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:45.478803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef233600-bc63-4e81-9078-b87df9e76dbf-crio-socket\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.478844 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.478797    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:02:45.478844 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.478809    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:45.97878862 +0000 UTC m=+46.675348181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:45.478928 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.478864    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:45.978846183 +0000 UTC m=+46.675405762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : configmap "kube-rbac-proxy" not found
Sep 15 12:02:45.478994 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.478979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef233600-bc63-4e81-9078-b87df9e76dbf-data-volume\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.486853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.486825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmg49\" (UniqueName: \"kubernetes.io/projected/eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5-kube-api-access-nmg49\") pod \"migrator-8b4d697c5-ph8vj\" (UID: \"eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"
Sep 15 12:02:45.487168 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.487146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m7kh\" (UniqueName: \"kubernetes.io/projected/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-api-access-5m7kh\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.506157 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.506129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:45.679414 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.679293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"
Sep 15 12:02:45.970134 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.970043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj"]
Sep 15 12:02:45.971342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.971311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:45.973328 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:45.973303    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a7ed2_ffa3_48d0_83f3_7601b4e458c5.slice/crio-049f057ad78d72270ae9efdfd50879c5ededcd1294e80025184d08745222386f WatchSource:0}: Error finding container 049f057ad78d72270ae9efdfd50879c5ededcd1294e80025184d08745222386f: Status 404 returned error can't find the container with id 049f057ad78d72270ae9efdfd50879c5ededcd1294e80025184d08745222386f
Sep 15 12:02:45.983885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.983858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.983991 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:45.983923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:45.984063 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.983992    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:45.984063 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.984041    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:02:45.984147 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.984073    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:46.984058458 +0000 UTC m=+47.680618013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:45.984200 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:45.984175    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:46.984165288 +0000 UTC m=+47.680724847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : configmap "kube-rbac-proxy" not found
Sep 15 12:02:46.047250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.047218    2583 generic.go:356] "Generic (PLEG): container finished" podID="15c02746-890f-4bfa-9bbe-c20980af3d45" containerID="c303bd41abf59d43ca18ec0ae6159325d22607450de64245ccb291a1c2d64f0a" exitCode=0
Sep 15 12:02:46.047651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.047299    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerDied","Data":"c303bd41abf59d43ca18ec0ae6159325d22607450de64245ccb291a1c2d64f0a"}
Sep 15 12:02:46.048578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.048557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj" event={"ID":"eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5","Type":"ContainerStarted","Data":"049f057ad78d72270ae9efdfd50879c5ededcd1294e80025184d08745222386f"}
Sep 15 12:02:46.050062 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.050045    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:02:46.050469 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.050452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/0.log"
Sep 15 12:02:46.050534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.050484    2583 generic.go:356] "Generic (PLEG): container finished" podID="c5570725-5835-48e5-8a10-599f37cffed7" containerID="7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f" exitCode=255
Sep 15 12:02:46.050534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.050521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" event={"ID":"c5570725-5835-48e5-8a10-599f37cffed7","Type":"ContainerDied","Data":"7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f"}
Sep 15 12:02:46.050641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.050564    2583 scope.go:117] "RemoveContainer" containerID="63ea1fcc22a19534e9c54564b9f11c3b2cda9c75a224d291b979a9fb1f9c05d1"
Sep 15 12:02:46.051415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.051396    2583 scope.go:117] "RemoveContainer" containerID="7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f"
Sep 15 12:02:46.051572 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:46.051553    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zw5m_openshift-console-operator(c5570725-5835-48e5-8a10-599f37cffed7)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podUID="c5570725-5835-48e5-8a10-599f37cffed7"
Sep 15 12:02:46.086138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.086099    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:46.263746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.263658    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:46.366994 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.366961    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:46.466975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.466940    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:46.658950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.658922    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xw4k7_144046aa-3d7c-4925-8259-da5480d92d3e/dns-node-resolver/0.log"
Sep 15 12:02:46.678562 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.678521    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:46.995008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.994920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:46.995008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:46.994972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:46.995259 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:46.995096    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:46.995259 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:46.995127    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:02:46.995259 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:46.995161    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:48.99514601 +0000 UTC m=+49.691705592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:46.995259 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:46.995184    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:48.995168695 +0000 UTC m=+49.691728253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : configmap "kube-rbac-proxy" not found
Sep 15 12:02:47.057855 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:47.057824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" event={"ID":"15c02746-890f-4bfa-9bbe-c20980af3d45","Type":"ContainerStarted","Data":"09ef48c77838b3a28b789b4589c024fac44eb451944a2ce628db9d91d855303b"}
Sep 15 12:02:47.059425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:47.059405    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:02:47.059718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:47.059701    2583 scope.go:117] "RemoveContainer" containerID="7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f"
Sep 15 12:02:47.059882 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:47.059862    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zw5m_openshift-console-operator(c5570725-5835-48e5-8a10-599f37cffed7)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podUID="c5570725-5835-48e5-8a10-599f37cffed7"
Sep 15 12:02:47.081681 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:47.081629    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9qdwb" podStartSLOduration=9.582703028 podStartE2EDuration="47.081616148s" totalImagesPullingTime="37.49891312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:47.080047118 +0000 UTC m=+47.776606704" watchObservedRunningTime="2026-09-15 12:02:47.081616148 +0000 UTC m=+47.778175716"
Sep 15 12:02:47.458122 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:47.458090    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f79s4_ea5ba13e-d405-4fb7-9030-811b949df8d8/node-ca/0.log"
Sep 15 12:02:48.057708 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.057672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zcjwh_81040b81-30da-48ef-963b-d42e89a16b58/serve-healthcheck-canary/0.log"
Sep 15 12:02:48.273384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.273351    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dq9th"]
Sep 15 12:02:48.275507 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.275487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.277975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.277951    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 12:02:48.278077 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.277998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 12:02:48.278895 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.278877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 12:02:48.279045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.278877    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wpvh9\""
Sep 15 12:02:48.279045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.278909    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 12:02:48.286071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.286050    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dq9th"]
Sep 15 12:02:48.306888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.306861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tghs\" (UniqueName: \"kubernetes.io/projected/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-kube-api-access-9tghs\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.306997 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.306896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-signing-key\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.307071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.307051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-signing-cabundle\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.407736 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.407687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tghs\" (UniqueName: \"kubernetes.io/projected/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-kube-api-access-9tghs\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.407736 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.407744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-signing-key\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.407930 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.407844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-signing-cabundle\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.408512 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.408491    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-signing-cabundle\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.410195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.410174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-signing-key\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.415602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.415578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tghs\" (UniqueName: \"kubernetes.io/projected/bc0f6bb1-af3d-402e-a59c-e9bf37c3220b-kube-api-access-9tghs\") pod \"service-ca-568869fdc5-dq9th\" (UID: \"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b\") " pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.584879 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.584841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-dq9th"
Sep 15 12:02:48.851306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:48.851280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dq9th"]
Sep 15 12:02:48.854449 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:02:48.854424    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0f6bb1_af3d_402e_a59c_e9bf37c3220b.slice/crio-b812c32dc5505f1559f9d30df5391e4e347203727cbcab9f9a508af487397c8e WatchSource:0}: Error finding container b812c32dc5505f1559f9d30df5391e4e347203727cbcab9f9a508af487397c8e: Status 404 returned error can't find the container with id b812c32dc5505f1559f9d30df5391e4e347203727cbcab9f9a508af487397c8e
Sep 15 12:02:49.012920 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.012872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:49.013140 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.012938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:49.013140 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.013088    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:49.013140 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.013111    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:02:49.013261 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.013173    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:53.013151685 +0000 UTC m=+53.709711257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:49.013261 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.013193    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:02:53.013183436 +0000 UTC m=+53.709742996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : configmap "kube-rbac-proxy" not found
Sep 15 12:02:49.066718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.066681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-dq9th" event={"ID":"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b","Type":"ContainerStarted","Data":"b1671f18eb1487e48de2a93ee497cc06082b7d7ef3e02e1f7fbfc3d6e49dbcfd"}
Sep 15 12:02:49.066884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.066737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-dq9th" event={"ID":"bc0f6bb1-af3d-402e-a59c-e9bf37c3220b","Type":"ContainerStarted","Data":"b812c32dc5505f1559f9d30df5391e4e347203727cbcab9f9a508af487397c8e"}
Sep 15 12:02:49.081854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.081806    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-dq9th" podStartSLOduration=1.081790211 podStartE2EDuration="1.081790211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:49.08070032 +0000 UTC m=+49.777259902" watchObservedRunningTime="2026-09-15 12:02:49.081790211 +0000 UTC m=+49.778349789"
Sep 15 12:02:49.618453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.618414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.618470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.618507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.618536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618592    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.618572625 +0000 UTC m=+66.315132205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : configmap references non-existent config key: service-ca.crt
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618657    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618670    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8495c67bfc-xtngb: secret "image-registry-tls" not found
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618669    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618714    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls podName:23e5d6de-da28-4ac9-b31d-544108426598 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.618700455 +0000 UTC m=+66.315260026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls") pod "image-registry-8495c67bfc-xtngb" (UID: "23e5d6de-da28-4ac9-b31d-544108426598") : secret "image-registry-tls" not found
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618740    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs podName:ef818076-7989-462f-b9f6-92a7cc8352d8 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.618720872 +0000 UTC m=+66.315280435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs") pod "router-default-847cffd544-fxqfj" (UID: "ef818076-7989-462f-b9f6-92a7cc8352d8") : secret "router-metrics-certs-default" not found
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618738    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:49.618872 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.618778    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls podName:da6c3092-9ef2-4a81-9a90-0deecff7c4bc nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.6187687 +0000 UTC m=+66.315328268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b5cnn" (UID: "da6c3092-9ef2-4a81-9a90-0deecff7c4bc") : secret "cluster-monitoring-operator-tls" not found
Sep 15 12:02:49.820934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.820897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:02:49.821134 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.821005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:02:49.821134 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.821087    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:02:49.821260 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.821166    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls podName:3b813070-33ae-426d-ad4a-1fba43dfc263 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.821145961 +0000 UTC m=+66.517705523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls") pod "dns-default-8ckhc" (UID: "3b813070-33ae-426d-ad4a-1fba43dfc263") : secret "dns-default-metrics-tls" not found
Sep 15 12:02:49.821260 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.821230    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 12:02:49.821358 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:49.821268    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls podName:9436cd56-d74c-4acf-b6bb-4ce6093ba6e1 nodeName:}" failed. No retries permitted until 2026-09-15 12:03:05.821257715 +0000 UTC m=+66.517817271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xcxvn" (UID: "9436cd56-d74c-4acf-b6bb-4ce6093ba6e1") : secret "samples-operator-tls" not found
Sep 15 12:02:49.881425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.881344    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:49.976489 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:49.976456    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.188358 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.188276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.287650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.287609    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.333717 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.333138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:02:50.333717 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:50.333293    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 12:02:50.333717 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:50.333361    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert podName:31a288ab-9731-41ec-8b6d-cc37c790fe6f nodeName:}" failed. No retries permitted until 2026-09-15 12:03:06.333341769 +0000 UTC m=+67.029901332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-lfzm7" (UID: "31a288ab-9731-41ec-8b6d-cc37c790fe6f") : secret "networking-console-plugin-cert" not found
Sep 15 12:02:50.408830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.408786    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.598950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.598897    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.696241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.696202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.818252 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.818222    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:50.963988 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:50.963954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:51.035249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:51.035214    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:51.077133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:51.077047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj" event={"ID":"eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5","Type":"ContainerStarted","Data":"00457767c65ed245a3f7a0c422e50dc3c6f35a78875e9106206b51f9dcd0eceb"}
Sep 15 12:02:51.077284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:51.077150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj" event={"ID":"eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5","Type":"ContainerStarted","Data":"e4586962a327b27f774a9ee4ee89f5a19434cd54c7f818a2e31261b1d18c4778"}
Sep 15 12:02:51.093595 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:51.093541    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-ph8vj" podStartSLOduration=1.573246481 podStartE2EDuration="6.093521304s" totalImagesPullingTime="4.520274823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:02:51.092488639 +0000 UTC m=+51.789048233" watchObservedRunningTime="2026-09-15 12:02:51.093521304 +0000 UTC m=+51.790080883"
Sep 15 12:02:51.278975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:51.278939    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:52.079065 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:52.079016    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:52.141513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:52.141477    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:53.059977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:53.059940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:53.060178 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:53.060148    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:02:53.060178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:53.060159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:02:53.060279 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:53.060197    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:03:01.060183244 +0000 UTC m=+61.756742801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : configmap "kube-rbac-proxy" not found
Sep 15 12:02:53.060279 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:53.060253    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:53.060391 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:53.060311    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls podName:ef233600-bc63-4e81-9078-b87df9e76dbf nodeName:}" failed. No retries permitted until 2026-09-15 12:03:01.060279839 +0000 UTC m=+61.756839394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4w7s9" (UID: "ef233600-bc63-4e81-9078-b87df9e76dbf") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:02:54.009303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:54.009264    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:54.009754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:54.009640    2583 scope.go:117] "RemoveContainer" containerID="7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f"
Sep 15 12:02:54.009809 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:54.009783    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zw5m_openshift-console-operator(c5570725-5835-48e5-8a10-599f37cffed7)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podUID="c5570725-5835-48e5-8a10-599f37cffed7"
Sep 15 12:02:54.141729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:54.141695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:02:54.142075 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:54.142061    2583 scope.go:117] "RemoveContainer" containerID="7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f"
Sep 15 12:02:54.142229 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:02:54.142212    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8zw5m_openshift-console-operator(c5570725-5835-48e5-8a10-599f37cffed7)\"" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" podUID="c5570725-5835-48e5-8a10-599f37cffed7"
Sep 15 12:02:55.615172 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:55.615134    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:55.718155 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:55.718118    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:55.967745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:55.967678    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:56.077036 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:56.077000    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:56.275756 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:56.275673    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:56.477817 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:56.477783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:56.677919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:56.677876    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:56.778449 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:56.778403    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:56.868268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:56.868236    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:58.068745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:58.068713    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:58.266208 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:58.266169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:58.469562 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:58.469465    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:58.672973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:58.672942    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:02:58.867821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:02:58.867781    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:01.056450 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.056424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrzkf"
Sep 15 12:03:01.129478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.129446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:03:01.129657 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.129531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:03:01.130183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.130158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef233600-bc63-4e81-9078-b87df9e76dbf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:03:01.131772 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.131753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef233600-bc63-4e81-9078-b87df9e76dbf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4w7s9\" (UID: \"ef233600-bc63-4e81-9078-b87df9e76dbf\") " pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:03:01.276923 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.276893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wlcl7\""
Sep 15 12:03:01.285728 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.285694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4w7s9"
Sep 15 12:03:01.570897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:01.570875    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4w7s9"]
Sep 15 12:03:01.573613 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:01.573586    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef233600_bc63_4e81_9078_b87df9e76dbf.slice/crio-146525b2fcfa3446004d25b47e8345cdae5c6585f87aec4e202fc19df8b64bc0 WatchSource:0}: Error finding container 146525b2fcfa3446004d25b47e8345cdae5c6585f87aec4e202fc19df8b64bc0: Status 404 returned error can't find the container with id 146525b2fcfa3446004d25b47e8345cdae5c6585f87aec4e202fc19df8b64bc0
Sep 15 12:03:02.112000 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:02.111964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4w7s9" event={"ID":"ef233600-bc63-4e81-9078-b87df9e76dbf","Type":"ContainerStarted","Data":"88c3411e05e34f41de50aa8a94ebecb08ef5b6551828fcafcfbda56016226903"}
Sep 15 12:03:02.112397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:02.112038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4w7s9" event={"ID":"ef233600-bc63-4e81-9078-b87df9e76dbf","Type":"ContainerStarted","Data":"146525b2fcfa3446004d25b47e8345cdae5c6585f87aec4e202fc19df8b64bc0"}
Sep 15 12:03:03.121104 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:03.121056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4w7s9" event={"ID":"ef233600-bc63-4e81-9078-b87df9e76dbf","Type":"ContainerStarted","Data":"4e39e54d55f60cee52a46505e008d5b7eaf4371df7d0402e09cbab228b88ea6e"}
Sep 15 12:03:03.221125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:03.221089    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:03.322272 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:03.322234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:04.126191 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:04.126154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4w7s9" event={"ID":"ef233600-bc63-4e81-9078-b87df9e76dbf","Type":"ContainerStarted","Data":"68f4e6619707157de3bff1a68d12b6eacf6807e81b4f3d0522ab7ac70801bf10"}
Sep 15 12:03:04.144867 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:04.144821    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4w7s9" podStartSLOduration=17.078823637 podStartE2EDuration="19.144805457s" totalImagesPullingTime="2.06598182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:04.143150499 +0000 UTC m=+64.839710088" watchObservedRunningTime="2026-09-15 12:03:04.144805457 +0000 UTC m=+64.841365031"
Sep 15 12:03:04.811628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:04.811588    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:04.886063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:04.886017    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:05.467483 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.467451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:03:05.467908 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.467512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:03:05.470092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.470074    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 12:03:05.470754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.470739    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 12:03:05.480649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.480621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cec4d24d-6cbf-485b-9754-3ab98fce5a6f-original-pull-secret\") pod \"global-pull-secret-syncer-66f9b\" (UID: \"cec4d24d-6cbf-485b-9754-3ab98fce5a6f\") " pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:03:05.480753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.480651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59f50a12-82fe-48ec-8b9f-26aa7c1133d4-metrics-certs\") pod \"network-metrics-daemon-g2n6z\" (UID: \"59f50a12-82fe-48ec-8b9f-26aa7c1133d4\") " pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:03:05.569004 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.568962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:03:05.571465 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.571439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tkg\" (UniqueName: \"kubernetes.io/projected/4e8e54dd-ad17-46b3-b460-0da6e0d0efef-kube-api-access-74tkg\") pod \"network-check-target-mjfj7\" (UID: \"4e8e54dd-ad17-46b3-b460-0da6e0d0efef\") " pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:03:05.669964 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.669913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:05.670160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.669975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:03:05.670160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.670013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:03:05.670160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.670121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:05.670827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.670794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef818076-7989-462f-b9f6-92a7cc8352d8-service-ca-bundle\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:05.672348 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.672322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6c3092-9ef2-4a81-9a90-0deecff7c4bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b5cnn\" (UID: \"da6c3092-9ef2-4a81-9a90-0deecff7c4bc\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:03:05.672446 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.672369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"image-registry-8495c67bfc-xtngb\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") " pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:03:05.672509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.672447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef818076-7989-462f-b9f6-92a7cc8352d8-metrics-certs\") pod \"router-default-847cffd544-fxqfj\" (UID: \"ef818076-7989-462f-b9f6-92a7cc8352d8\") " pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:05.692460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.692432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-66f9b"
Sep 15 12:03:05.713341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.713311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l82m\""
Sep 15 12:03:05.717103 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.717082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k48ps\""
Sep 15 12:03:05.721879 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.721832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2n6z"
Sep 15 12:03:05.725675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.725657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:03:05.844367 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.844334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s8xfw\""
Sep 15 12:03:05.852369 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.852345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:03:05.863146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.863124    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ht528\""
Sep 15 12:03:05.871305 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.871276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:05.871564 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.871546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:03:05.871630 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.871589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:03:05.875505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.875481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b813070-33ae-426d-ad4a-1fba43dfc263-metrics-tls\") pod \"dns-default-8ckhc\" (UID: \"3b813070-33ae-426d-ad4a-1fba43dfc263\") " pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:03:05.875658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.875636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9436cd56-d74c-4acf-b6bb-4ce6093ba6e1-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xcxvn\" (UID: \"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:03:05.888807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.888780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-59pzv\""
Sep 15 12:03:05.896873 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.896835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"
Sep 15 12:03:05.966645 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:05.966614    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-66f9b"]
Sep 15 12:03:06.002120 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.001987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g2n6z"]
Sep 15 12:03:06.004524 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:06.004498    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f50a12_82fe_48ec_8b9f_26aa7c1133d4.slice/crio-39eca4e2333085448cc75d6535bb008c1619f51e8f81f6fa2be3cfa3c1803878 WatchSource:0}: Error finding container 39eca4e2333085448cc75d6535bb008c1619f51e8f81f6fa2be3cfa3c1803878: Status 404 returned error can't find the container with id 39eca4e2333085448cc75d6535bb008c1619f51e8f81f6fa2be3cfa3c1803878
Sep 15 12:03:06.007044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.007005    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mjfj7"]
Sep 15 12:03:06.009437 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:06.009415    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8e54dd_ad17_46b3_b460_0da6e0d0efef.slice/crio-7da23e2126dd329714f195d12a96f2fcf780ca05c656ee114ce6420bc46f676d WatchSource:0}: Error finding container 7da23e2126dd329714f195d12a96f2fcf780ca05c656ee114ce6420bc46f676d: Status 404 returned error can't find the container with id 7da23e2126dd329714f195d12a96f2fcf780ca05c656ee114ce6420bc46f676d
Sep 15 12:03:06.010397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.010378    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lcgbm\""
Sep 15 12:03:06.018524 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.018504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"
Sep 15 12:03:06.033364 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.033343    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gwv7\""
Sep 15 12:03:06.042862 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.041799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:03:06.126501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.126469    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8495c67bfc-xtngb"]
Sep 15 12:03:06.128891 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:06.128857    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e5d6de_da28_4ac9_b31d_544108426598.slice/crio-0a5a569b9c8079fef98a59e79b1812b9c376b343a93b171ff1c2300209c2f663 WatchSource:0}: Error finding container 0a5a569b9c8079fef98a59e79b1812b9c376b343a93b171ff1c2300209c2f663: Status 404 returned error can't find the container with id 0a5a569b9c8079fef98a59e79b1812b9c376b343a93b171ff1c2300209c2f663
Sep 15 12:03:06.134083 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.134006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-66f9b" event={"ID":"cec4d24d-6cbf-485b-9754-3ab98fce5a6f","Type":"ContainerStarted","Data":"e62e4332a5795e9892cb7667924037d6ec91b73900bcb0f5c60b98e9bcecbe41"}
Sep 15 12:03:06.135604 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.135581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mjfj7" event={"ID":"4e8e54dd-ad17-46b3-b460-0da6e0d0efef","Type":"ContainerStarted","Data":"1a64040ada50fc3cabecba581cebc61d1462bf8d6a9b2b107ef02662b2056e19"}
Sep 15 12:03:06.135708 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.135627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mjfj7" event={"ID":"4e8e54dd-ad17-46b3-b460-0da6e0d0efef","Type":"ContainerStarted","Data":"7da23e2126dd329714f195d12a96f2fcf780ca05c656ee114ce6420bc46f676d"}
Sep 15 12:03:06.135708 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.135684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:03:06.136866 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.136841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g2n6z" event={"ID":"59f50a12-82fe-48ec-8b9f-26aa7c1133d4","Type":"ContainerStarted","Data":"39eca4e2333085448cc75d6535bb008c1619f51e8f81f6fa2be3cfa3c1803878"}
Sep 15 12:03:06.137867 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.137849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" event={"ID":"23e5d6de-da28-4ac9-b31d-544108426598","Type":"ContainerStarted","Data":"0a5a569b9c8079fef98a59e79b1812b9c376b343a93b171ff1c2300209c2f663"}
Sep 15 12:03:06.149469 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.149447    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-847cffd544-fxqfj"]
Sep 15 12:03:06.151376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.151338    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mjfj7" podStartSLOduration=66.15132477 podStartE2EDuration="1m6.15132477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:06.15080073 +0000 UTC m=+66.847360339" watchObservedRunningTime="2026-09-15 12:03:06.15132477 +0000 UTC m=+66.847884347"
Sep 15 12:03:06.165064 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:06.165011    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef818076_7989_462f_b9f6_92a7cc8352d8.slice/crio-a08d5e1bc665909ce243f1b24b7fd2b6e4fbb07a34d4184785d305d24562bbb6 WatchSource:0}: Error finding container a08d5e1bc665909ce243f1b24b7fd2b6e4fbb07a34d4184785d305d24562bbb6: Status 404 returned error can't find the container with id a08d5e1bc665909ce243f1b24b7fd2b6e4fbb07a34d4184785d305d24562bbb6
Sep 15 12:03:06.181352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.181281    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn"]
Sep 15 12:03:06.183318 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:06.183297    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6c3092_9ef2_4a81_9a90_0deecff7c4bc.slice/crio-3fd69d30eb4a9d7f06b34966f1a9b65881e86e23f5318f04c591b5a1c3925433 WatchSource:0}: Error finding container 3fd69d30eb4a9d7f06b34966f1a9b65881e86e23f5318f04c591b5a1c3925433: Status 404 returned error can't find the container with id 3fd69d30eb4a9d7f06b34966f1a9b65881e86e23f5318f04c591b5a1c3925433
Sep 15 12:03:06.286189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.286135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn"]
Sep 15 12:03:06.320060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.320007    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8ckhc"]
Sep 15 12:03:06.333935 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:06.333912    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b813070_33ae_426d_ad4a_1fba43dfc263.slice/crio-6d75e141a0d399b51e3e22c6e9c1fe91015d8599b236fa9539bbec894ceeabd5 WatchSource:0}: Error finding container 6d75e141a0d399b51e3e22c6e9c1fe91015d8599b236fa9539bbec894ceeabd5: Status 404 returned error can't find the container with id 6d75e141a0d399b51e3e22c6e9c1fe91015d8599b236fa9539bbec894ceeabd5
Sep 15 12:03:06.375464 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.375426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:03:06.377954 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.377928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/31a288ab-9731-41ec-8b6d-cc37c790fe6f-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lfzm7\" (UID: \"31a288ab-9731-41ec-8b6d-cc37c790fe6f\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:03:06.588945 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.588912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-45459\""
Sep 15 12:03:06.597101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.597066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"
Sep 15 12:03:06.885870 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:06.885774    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7"]
Sep 15 12:03:07.118893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.118837    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:07.145789 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.145653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn" event={"ID":"da6c3092-9ef2-4a81-9a90-0deecff7c4bc","Type":"ContainerStarted","Data":"3fd69d30eb4a9d7f06b34966f1a9b65881e86e23f5318f04c591b5a1c3925433"}
Sep 15 12:03:07.148441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.148413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn" event={"ID":"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1","Type":"ContainerStarted","Data":"b0491577956c2e32f4c12614b1ee6c592190bdc294f44865e62a70e95c0b9f99"}
Sep 15 12:03:07.150703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.150612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" event={"ID":"23e5d6de-da28-4ac9-b31d-544108426598","Type":"ContainerStarted","Data":"4353ae2d072dc217c143b1e2a55f7780367ca00a982b8d3bb430401b3f592c2a"}
Sep 15 12:03:07.150886 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.150866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:03:07.153017 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.152990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-847cffd544-fxqfj" event={"ID":"ef818076-7989-462f-b9f6-92a7cc8352d8","Type":"ContainerStarted","Data":"5b7d022a4eb1bfb108e505e71e867ec21dfb12f69ec853c0a224a4c8c03ee9bb"}
Sep 15 12:03:07.153142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.153064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-847cffd544-fxqfj" event={"ID":"ef818076-7989-462f-b9f6-92a7cc8352d8","Type":"ContainerStarted","Data":"a08d5e1bc665909ce243f1b24b7fd2b6e4fbb07a34d4184785d305d24562bbb6"}
Sep 15 12:03:07.155288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.155236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7" event={"ID":"31a288ab-9731-41ec-8b6d-cc37c790fe6f","Type":"ContainerStarted","Data":"cb6c41f014f7de8d0670503640de279bf695ad547a3312f5e191a777d7d772e9"}
Sep 15 12:03:07.156904 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.156836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8ckhc" event={"ID":"3b813070-33ae-426d-ad4a-1fba43dfc263","Type":"ContainerStarted","Data":"6d75e141a0d399b51e3e22c6e9c1fe91015d8599b236fa9539bbec894ceeabd5"}
Sep 15 12:03:07.172450 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.171473    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" podStartSLOduration=67.171454315 podStartE2EDuration="1m7.171454315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:07.170012371 +0000 UTC m=+67.866571966" watchObservedRunningTime="2026-09-15 12:03:07.171454315 +0000 UTC m=+67.868013894"
Sep 15 12:03:07.176249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.176141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:07.187934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.187886    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-847cffd544-fxqfj" podStartSLOduration=42.187870998 podStartE2EDuration="42.187870998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:07.187185291 +0000 UTC m=+67.883744872" watchObservedRunningTime="2026-09-15 12:03:07.187870998 +0000 UTC m=+67.884430557"
Sep 15 12:03:07.872291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.872259    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:07.875863 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:07.875831    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:08.163193 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:08.163107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g2n6z" event={"ID":"59f50a12-82fe-48ec-8b9f-26aa7c1133d4","Type":"ContainerStarted","Data":"05f83fbdea18ab75685355ea5284c1df146ff424753f033f866e83eac5f9cb6e"}
Sep 15 12:03:08.163378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:08.163357    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:08.164873 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:08.164846    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-847cffd544-fxqfj"
Sep 15 12:03:08.800790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:08.800705    2583 scope.go:117] "RemoveContainer" containerID="7d34253314884e70ce642b44d371daefc942359e67550301c280dc9e6383260f"
Sep 15 12:03:09.495657 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.495624    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"]
Sep 15 12:03:09.511675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.511589    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:09.527751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.527637    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"]
Sep 15 12:03:09.527897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.527876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.530734 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.530699    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 12:03:09.531011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.530991    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 12:03:09.531142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.531079    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 15 12:03:09.531457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.531437    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tn9qt\""
Sep 15 12:03:09.531558 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.531487    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 12:03:09.543675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.543647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"]
Sep 15 12:03:09.543789 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.543769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.546118 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.546097    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 12:03:09.565971 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.565947    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"]
Sep 15 12:03:09.565971 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.565975    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"]
Sep 15 12:03:09.566132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.565996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"]
Sep 15 12:03:09.566132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.566086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.569062 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.569043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 12:03:09.569250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.569234    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 12:03:09.569615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.569601    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 12:03:09.569922 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.569905    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 12:03:09.608397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.608366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw469\" (UniqueName: \"kubernetes.io/projected/b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00-kube-api-access-dw469\") pod \"managed-serviceaccount-addon-agent-58b4778546-sfns2\" (UID: \"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.608517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.608413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58b4778546-sfns2\" (UID: \"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.633144 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.633116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:09.709381 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/52484643-838d-47cf-9566-fc8d30bf3acb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.709509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/065f282d-5942-4182-9c8a-f03fb3159f85-tmp\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.709509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65xp\" (UniqueName: \"kubernetes.io/projected/065f282d-5942-4182-9c8a-f03fb3159f85-kube-api-access-r65xp\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.709509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-ca\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.709509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.709651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-hub\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.709651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw469\" (UniqueName: \"kubernetes.io/projected/b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00-kube-api-access-dw469\") pod \"managed-serviceaccount-addon-agent-58b4778546-sfns2\" (UID: \"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.709651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58b4778546-sfns2\" (UID: \"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.709651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.709766 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflks\" (UniqueName: \"kubernetes.io/projected/52484643-838d-47cf-9566-fc8d30bf3acb-kube-api-access-nflks\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.709766 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.709668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/065f282d-5942-4182-9c8a-f03fb3159f85-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.712205 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.712178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58b4778546-sfns2\" (UID: \"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.722648 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.722626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw469\" (UniqueName: \"kubernetes.io/projected/b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00-kube-api-access-dw469\") pod \"managed-serviceaccount-addon-agent-58b4778546-sfns2\" (UID: \"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.767226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.767146    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:09.810273 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.810273 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflks\" (UniqueName: \"kubernetes.io/projected/52484643-838d-47cf-9566-fc8d30bf3acb-kube-api-access-nflks\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.810474 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/065f282d-5942-4182-9c8a-f03fb3159f85-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.810474 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/52484643-838d-47cf-9566-fc8d30bf3acb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.810474 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/065f282d-5942-4182-9c8a-f03fb3159f85-tmp\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.810627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65xp\" (UniqueName: \"kubernetes.io/projected/065f282d-5942-4182-9c8a-f03fb3159f85-kube-api-access-r65xp\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.810627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-ca\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.810627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.810627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-hub\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.810817 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.810769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/065f282d-5942-4182-9c8a-f03fb3159f85-tmp\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.811129 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.811104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/52484643-838d-47cf-9566-fc8d30bf3acb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.813141 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.813110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-hub\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.813347 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.813327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.813414 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.813391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-ca\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.813496 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.813478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/065f282d-5942-4182-9c8a-f03fb3159f85-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.813619 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.813600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52484643-838d-47cf-9566-fc8d30bf3acb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.819240 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.819221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65xp\" (UniqueName: \"kubernetes.io/projected/065f282d-5942-4182-9c8a-f03fb3159f85-kube-api-access-r65xp\") pod \"klusterlet-addon-workmgr-5bfd577ff7-7tpd9\" (UID: \"065f282d-5942-4182-9c8a-f03fb3159f85\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.819827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.819810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflks\" (UniqueName: \"kubernetes.io/projected/52484643-838d-47cf-9566-fc8d30bf3acb-kube-api-access-nflks\") pod \"cluster-proxy-proxy-agent-67696c869c-znvrz\" (UID: \"52484643-838d-47cf-9566-fc8d30bf3acb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:09.846228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.846199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"
Sep 15 12:03:09.854357 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.854334    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:09.868188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.868164    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:09.874931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:09.874913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"
Sep 15 12:03:11.796115 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:11.795954    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz"]
Sep 15 12:03:11.808175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:11.808116    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"]
Sep 15 12:03:11.819570 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:11.819543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2"]
Sep 15 12:03:11.968075 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:11.967866    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.065757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.065717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.141399 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.141193    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"]
Sep 15 12:03:12.144204 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.144105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.162324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.161103    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"]
Sep 15 12:03:12.162324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.161221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:12.164735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.164185    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bc45f\""
Sep 15 12:03:12.164735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.164410    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 12:03:12.183967 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.183935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9" event={"ID":"065f282d-5942-4182-9c8a-f03fb3159f85","Type":"ContainerStarted","Data":"dbda1ff3d0edb333ba9ac1afac2b069c3504478b11a6f873c8387b2f2e9242fc"}
Sep 15 12:03:12.190995 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.190721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-66f9b" event={"ID":"cec4d24d-6cbf-485b-9754-3ab98fce5a6f","Type":"ContainerStarted","Data":"07a3cdd3464f7f2232e58b22a1d67410e21003f918165d9309e2c48a4f8fcff1"}
Sep 15 12:03:12.193068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.192993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7" event={"ID":"31a288ab-9731-41ec-8b6d-cc37c790fe6f","Type":"ContainerStarted","Data":"6dec95b5a146a252adf50846d16174bd1ee715e4165cdb98ae7c35e3e3282c34"}
Sep 15 12:03:12.196153 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.196122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g2n6z" event={"ID":"59f50a12-82fe-48ec-8b9f-26aa7c1133d4","Type":"ContainerStarted","Data":"bd852a0fe02e390bc4adf7417ae9baa6e399ea3d230e02533d281d1f533c0e70"}
Sep 15 12:03:12.198838 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.198814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz" event={"ID":"52484643-838d-47cf-9566-fc8d30bf3acb","Type":"ContainerStarted","Data":"95e3b02457a83adeece00071b3ea69d39fe537021d2fbb2b3d5ee4e3693639e0"}
Sep 15 12:03:12.201242 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.201223    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:03:12.201367 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.201349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m" event={"ID":"c5570725-5835-48e5-8a10-599f37cffed7","Type":"ContainerStarted","Data":"ff30462c60667a2b12c2af896c6f4d05771ff7226117f289e36ab8b343037cf0"}
Sep 15 12:03:12.201673 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.201633    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:03:12.203372 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.203285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2" event={"ID":"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00","Type":"ContainerStarted","Data":"ae5246fc33579b1cb454748dd6cb1839bc206d3d626e66eb871b16984f6bdf24"}
Sep 15 12:03:12.207122 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.205953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn" event={"ID":"da6c3092-9ef2-4a81-9a90-0deecff7c4bc","Type":"ContainerStarted","Data":"a20784b9baefb0e11954f7a60cdc0939c4d82181eef60c3002ea1101f2478f9a"}
Sep 15 12:03:12.207122 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.206557    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-66f9b" podStartSLOduration=66.270401396 podStartE2EDuration="1m12.2065411s" totalImagesPullingTime="5.936139704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:12.204977601 +0000 UTC m=+72.901537195" watchObservedRunningTime="2026-09-15 12:03:12.2065411 +0000 UTC m=+72.903100669"
Sep 15 12:03:12.210344 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.210322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn" event={"ID":"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1","Type":"ContainerStarted","Data":"abcdd51a9c020fd1a7e98dc44dacad3b83261d2e0b69a107edd989be08f4a373"}
Sep 15 12:03:12.210993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.210974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn" event={"ID":"9436cd56-d74c-4acf-b6bb-4ce6093ba6e1","Type":"ContainerStarted","Data":"b51ab43af6f72a1fb11b08cc4d37af498b4940609149bf6eb107b91a9e92d590"}
Sep 15 12:03:12.222681 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.222642    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lfzm7" podStartSLOduration=33.607287906 podStartE2EDuration="38.22262822s" totalImagesPullingTime="4.615340314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:12.222001265 +0000 UTC m=+72.918560868" watchObservedRunningTime="2026-09-15 12:03:12.22262822 +0000 UTC m=+72.919187795"
Sep 15 12:03:12.274050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.272965    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b5cnn" podStartSLOduration=41.948235414 podStartE2EDuration="47.272943518s" totalImagesPullingTime="5.324708104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:12.271670062 +0000 UTC m=+72.968229672" watchObservedRunningTime="2026-09-15 12:03:12.272943518 +0000 UTC m=+72.969503098"
Sep 15 12:03:12.274050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.273404    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g2n6z" podStartSLOduration=70.453914166 podStartE2EDuration="1m12.27339194s" totalImagesPullingTime="1.819477774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:12.256855708 +0000 UTC m=+72.953415287" watchObservedRunningTime="2026-09-15 12:03:12.27339194 +0000 UTC m=+72.969951519"
Sep 15 12:03:12.291850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.291404    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xcxvn" podStartSLOduration=42.1378187 podStartE2EDuration="47.291018458s" totalImagesPullingTime="5.153199758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:12.289297513 +0000 UTC m=+72.985857092" watchObservedRunningTime="2026-09-15 12:03:12.291018458 +0000 UTC m=+72.987578039"
Sep 15 12:03:12.336186 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.336003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4017423e-1b35-4c92-ae30-65a8c32fc94b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cw7jn\" (UID: \"4017423e-1b35-4c92-ae30-65a8c32fc94b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:12.353515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.353478    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.436764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.436725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4017423e-1b35-4c92-ae30-65a8c32fc94b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cw7jn\" (UID: \"4017423e-1b35-4c92-ae30-65a8c32fc94b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:12.440193 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.440163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4017423e-1b35-4c92-ae30-65a8c32fc94b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cw7jn\" (UID: \"4017423e-1b35-4c92-ae30-65a8c32fc94b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:12.452102 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.451997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.467954 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.467926    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8zw5m"
Sep 15 12:03:12.476005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.475976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:12.552833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.552757    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.646685 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.646598    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-xmttb"]
Sep 15 12:03:12.655331 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.655127    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.677820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.677788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-xmttb"]
Sep 15 12:03:12.677977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.677876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:12.680587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.680562    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 12:03:12.680780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.680762    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 12:03:12.681593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.681570    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-z8scl\""
Sep 15 12:03:12.717135 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.717098    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.776460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.776420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.785478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.785378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"]
Sep 15 12:03:12.790137 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:12.790104    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4017423e_1b35_4c92_ae30_65a8c32fc94b.slice/crio-f7df62c37e43557702f14beb3cd3f479a52ee026e939ca5a91abfd2fffc7a2bf WatchSource:0}: Error finding container f7df62c37e43557702f14beb3cd3f479a52ee026e939ca5a91abfd2fffc7a2bf: Status 404 returned error can't find the container with id f7df62c37e43557702f14beb3cd3f479a52ee026e939ca5a91abfd2fffc7a2bf
Sep 15 12:03:12.843386 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.843302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zlmd\" (UniqueName: \"kubernetes.io/projected/d93ac03f-b94b-4d6f-b8e4-857222a82144-kube-api-access-5zlmd\") pod \"downloads-76f99584f4-xmttb\" (UID: \"d93ac03f-b94b-4d6f-b8e4-857222a82144\") " pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:12.929734 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.929627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.944342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.944223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zlmd\" (UniqueName: \"kubernetes.io/projected/d93ac03f-b94b-4d6f-b8e4-857222a82144-kube-api-access-5zlmd\") pod \"downloads-76f99584f4-xmttb\" (UID: \"d93ac03f-b94b-4d6f-b8e4-857222a82144\") " pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:12.954217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.954154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zlmd\" (UniqueName: \"kubernetes.io/projected/d93ac03f-b94b-4d6f-b8e4-857222a82144-kube-api-access-5zlmd\") pod \"downloads-76f99584f4-xmttb\" (UID: \"d93ac03f-b94b-4d6f-b8e4-857222a82144\") " pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:12.985519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.985486    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:12.990547 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:12.990276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:13.145285 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:13.145231    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:13.219919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:13.219290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn" event={"ID":"4017423e-1b35-4c92-ae30-65a8c32fc94b","Type":"ContainerStarted","Data":"f7df62c37e43557702f14beb3cd3f479a52ee026e939ca5a91abfd2fffc7a2bf"}
Sep 15 12:03:13.325671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:13.325618    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:13.423584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:13.423520    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:14.141847 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:14.141803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:14.333890 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:14.333806    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:14.454772 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:14.454674    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:14.556388 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:14.556264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:14.768073 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:14.767978    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:14.829208 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:14.829003    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:15.280488 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:15.280453    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:15.380469 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:15.380401    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:15.707385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:15.707348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:15.761748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:15.761710    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:18.449722 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:18.449677    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:18.527117 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:18.527017    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-xmttb"]
Sep 15 12:03:18.530405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:18.530374    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:18.862313 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:18.862270    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93ac03f_b94b_4d6f_b8e4_857222a82144.slice/crio-5f637ee11237fad4df2156d69e8660ac9828fb0a0b9ec318f861c402578ac756 WatchSource:0}: Error finding container 5f637ee11237fad4df2156d69e8660ac9828fb0a0b9ec318f861c402578ac756: Status 404 returned error can't find the container with id 5f637ee11237fad4df2156d69e8660ac9828fb0a0b9ec318f861c402578ac756
Sep 15 12:03:19.247346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.247258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-xmttb" event={"ID":"d93ac03f-b94b-4d6f-b8e4-857222a82144","Type":"ContainerStarted","Data":"5f637ee11237fad4df2156d69e8660ac9828fb0a0b9ec318f861c402578ac756"}
Sep 15 12:03:19.248615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.248589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9" event={"ID":"065f282d-5942-4182-9c8a-f03fb3159f85","Type":"ContainerStarted","Data":"da1e8cf230b58e5bb1d8a9166c80a4948c64172a6cb59e4ffdce853490d4c584"}
Sep 15 12:03:19.248837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.248816    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:19.250367 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.250333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn" event={"ID":"4017423e-1b35-4c92-ae30-65a8c32fc94b","Type":"ContainerStarted","Data":"ca59a342dd135f6f8875b2cd36bc123defa9611df4df7937f1892b9b490e0649"}
Sep 15 12:03:19.250472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.250421    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9"
Sep 15 12:03:19.250472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.250442    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:19.252667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.252644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz" event={"ID":"52484643-838d-47cf-9566-fc8d30bf3acb","Type":"ContainerStarted","Data":"d40d46a0efc20473d4613312eb496ff06fe77855ca438c3d6a8eb3a0e4a978ae"}
Sep 15 12:03:19.252774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.252674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz" event={"ID":"52484643-838d-47cf-9566-fc8d30bf3acb","Type":"ContainerStarted","Data":"4eaae341b56e2f2e4473009c13bca7e2ba7418f8f81940e9786227cc31dcd8b0"}
Sep 15 12:03:19.252774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.252694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz" event={"ID":"52484643-838d-47cf-9566-fc8d30bf3acb","Type":"ContainerStarted","Data":"c9a385747abbff7985837a9ba3ab679338018b8000cf56b1e6d7f75e9d0a4c27"}
Sep 15 12:03:19.254506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.254487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8ckhc" event={"ID":"3b813070-33ae-426d-ad4a-1fba43dfc263","Type":"ContainerStarted","Data":"df2a6403c18099cb1d4ffa128103b5b50d9ee0191678a2abd3b947f5fe5ece96"}
Sep 15 12:03:19.254607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.254524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8ckhc" event={"ID":"3b813070-33ae-426d-ad4a-1fba43dfc263","Type":"ContainerStarted","Data":"019ae92f2edbbd866bcd907811d1d6d5c09ce1fbc6c4abb12c32935055d7f211"}
Sep 15 12:03:19.254607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.254564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:03:19.255549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.255531    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn"
Sep 15 12:03:19.256102 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.256084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2" event={"ID":"b34b4b3c-0b44-4d59-b0b9-c7e9dab1dd00","Type":"ContainerStarted","Data":"b0a79b5597b34c3229fddeb5eb8f8966236dfa1fe16c34eb7e02ac296eadd764"}
Sep 15 12:03:19.264842 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.264796    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bfd577ff7-7tpd9" podStartSLOduration=3.257049295 podStartE2EDuration="10.26478286s" totalImagesPullingTime="7.007733565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.263508092 +0000 UTC m=+79.960067699" watchObservedRunningTime="2026-09-15 12:03:19.26478286 +0000 UTC m=+79.961342437"
Sep 15 12:03:19.278958 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.278912    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8ckhc" podStartSLOduration=34.337099374 podStartE2EDuration="46.278895593s" totalImagesPullingTime="11.941796219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.277955509 +0000 UTC m=+79.974515097" watchObservedRunningTime="2026-09-15 12:03:19.278895593 +0000 UTC m=+79.975455171"
Sep 15 12:03:19.312374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.312312    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67696c869c-znvrz" podStartSLOduration=3.309246917 podStartE2EDuration="10.31229125s" totalImagesPullingTime="7.003044333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.311406173 +0000 UTC m=+80.007965752" watchObservedRunningTime="2026-09-15 12:03:19.31229125 +0000 UTC m=+80.008850829"
Sep 15 12:03:19.326699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.326631    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cw7jn" podStartSLOduration=1.21612382 podStartE2EDuration="7.326612798s" totalImagesPullingTime="6.110488978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.32553262 +0000 UTC m=+80.022092210" watchObservedRunningTime="2026-09-15 12:03:19.326612798 +0000 UTC m=+80.023172375"
Sep 15 12:03:19.342268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:19.342214    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58b4778546-sfns2" podStartSLOduration=3.332206685 podStartE2EDuration="10.342195763s" totalImagesPullingTime="7.009989078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:19.339955833 +0000 UTC m=+80.036515410" watchObservedRunningTime="2026-09-15 12:03:19.342195763 +0000 UTC m=+80.038755368"
Sep 15 12:03:20.162374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:20.162322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:20.229298 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:20.229254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:20.411562 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:20.411525    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:22.116389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:22.116239    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:22.217717 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:22.217674    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:22.445676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:22.445589    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:22.545945 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:22.545905    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:22.646692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:22.646647    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.310090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.310050    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.530144 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.530041    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.577042 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.576993    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j45ww"]
Sep 15 12:03:24.580458 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.580434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.582973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.582948    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 12:03:24.585725 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.585686    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 12:03:24.586612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.586574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bh5pf\""
Sep 15 12:03:24.587092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.587071    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 12:03:24.587320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.587305    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 12:03:24.633241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.633205    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.648306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-wtmp\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-root\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47328fc0-2033-4a89-8479-01152dfcf1e8-metrics-client-ca\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-sys\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66kn\" (UniqueName: \"kubernetes.io/projected/47328fc0-2033-4a89-8479-01152dfcf1e8-kube-api-access-t66kn\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-textfile\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.648796 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.648685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-tls\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.694526 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.694486    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.749834 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.749752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-textfile\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.749849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-tls\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.749907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-wtmp\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.749940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-root\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.749967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47328fc0-2033-4a89-8479-01152dfcf1e8-metrics-client-ca\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-sys\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66kn\" (UniqueName: \"kubernetes.io/projected/47328fc0-2033-4a89-8479-01152dfcf1e8-kube-api-access-t66kn\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-root\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-textfile\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-wtmp\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47328fc0-2033-4a89-8479-01152dfcf1e8-sys\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.750911 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.750867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.751190 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.751163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47328fc0-2033-4a89-8479-01152dfcf1e8-metrics-client-ca\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.752662 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.752633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.752755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.752739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47328fc0-2033-4a89-8479-01152dfcf1e8-node-exporter-tls\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.760889 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.760856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66kn\" (UniqueName: \"kubernetes.io/projected/47328fc0-2033-4a89-8479-01152dfcf1e8-kube-api-access-t66kn\") pod \"node-exporter-j45ww\" (UID: \"47328fc0-2033-4a89-8479-01152dfcf1e8\") " pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.793362 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.793327    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.894100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.893336    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:24.897635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.897602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j45ww"
Sep 15 12:03:24.912702 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:03:24.912661    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47328fc0_2033_4a89_8479_01152dfcf1e8.slice/crio-f619fed93b24abdd4a802d435ec42e681268dd546590058f484ef607c7b266a2 WatchSource:0}: Error finding container f619fed93b24abdd4a802d435ec42e681268dd546590058f484ef607c7b266a2: Status 404 returned error can't find the container with id f619fed93b24abdd4a802d435ec42e681268dd546590058f484ef607c7b266a2
Sep 15 12:03:24.994755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:24.994719    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.055572 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.055522    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.259065 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.258952    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.286901 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.286866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j45ww" event={"ID":"47328fc0-2033-4a89-8479-01152dfcf1e8","Type":"ContainerStarted","Data":"f619fed93b24abdd4a802d435ec42e681268dd546590058f484ef607c7b266a2"}
Sep 15 12:03:25.360053 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.359949    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.535704 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.535626    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.616663 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.616582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.688606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.688567    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:25.858559 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.858514    2583 patch_prober.go:28] interesting pod/image-registry-8495c67bfc-xtngb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 12:03:25.858737 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:25.858588    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:03:26.293385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:26.293288    2583 generic.go:356] "Generic (PLEG): container finished" podID="47328fc0-2033-4a89-8479-01152dfcf1e8" containerID="2daf0584b43a3c0f50ca56d934ede59ed88adef3381330cca809557729d817db" exitCode=0
Sep 15 12:03:26.293547 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:26.293390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j45ww" event={"ID":"47328fc0-2033-4a89-8479-01152dfcf1e8","Type":"ContainerDied","Data":"2daf0584b43a3c0f50ca56d934ede59ed88adef3381330cca809557729d817db"}
Sep 15 12:03:27.302815 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.302781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j45ww" event={"ID":"47328fc0-2033-4a89-8479-01152dfcf1e8","Type":"ContainerStarted","Data":"d1c09f2c6fae28ac6766b947af897c27df33a213d2849f866778ec9e98e1cb9d"}
Sep 15 12:03:27.303294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.302844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j45ww" event={"ID":"47328fc0-2033-4a89-8479-01152dfcf1e8","Type":"ContainerStarted","Data":"214cdceef995bb03c9c54f38981a46f294553cb1bd9d708fec15ddfab6a09d5b"}
Sep 15 12:03:27.321530 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.321472    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j45ww" podStartSLOduration=2.262237612 podStartE2EDuration="3.321452013s" totalImagesPullingTime="984.721482ms" totalInitContainerRuntime="74.492919ms" isStatefulPod=true podCreationTimestamp="2026-09-15 12:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:27.320803957 +0000 UTC m=+88.017363559" watchObservedRunningTime="2026-09-15 12:03:27.321452013 +0000 UTC m=+88.018011592"
Sep 15 12:03:27.465741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.465682    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:27.548466 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.548420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:27.753517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.753451    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:27.958186 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:27.958144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:28.157200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:28.157165    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:28.169823 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:28.169786    2583 patch_prober.go:28] interesting pod/image-registry-8495c67bfc-xtngb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 12:03:28.169974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:28.169847    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:03:29.001256 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:29.001155    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:29.080753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:29.080712    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:29.264960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:29.264868    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8ckhc"
Sep 15 12:03:29.326969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:29.326891    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:29.422543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:29.422476    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:29.674629 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:29.674583    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:30.769482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:30.769444    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:32.399635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:32.399586    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:32.502217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:32.502181    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:35.053936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:35.053895    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:35.119474 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:35.119434    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:35.857843 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:35.857800    2583 patch_prober.go:28] interesting pod/image-registry-8495c67bfc-xtngb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 12:03:35.858018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:35.857884    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:03:36.348514 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:36.348471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-xmttb" event={"ID":"d93ac03f-b94b-4d6f-b8e4-857222a82144","Type":"ContainerStarted","Data":"bed43865047b19da7e0fa616156d40c0f8750aca45f9fa6e5338022442613ade"}
Sep 15 12:03:36.348995 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:36.348655    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:36.361636 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:36.361602    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-xmttb"
Sep 15 12:03:36.367098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:36.367036    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-xmttb" podStartSLOduration=7.383680089 podStartE2EDuration="24.367000879s" totalImagesPullingTime="16.98332079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:03:36.365276627 +0000 UTC m=+97.061836242" watchObservedRunningTime="2026-09-15 12:03:36.367000879 +0000 UTC m=+97.063560457"
Sep 15 12:03:37.160265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:37.160232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mjfj7"
Sep 15 12:03:37.220039 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:37.219980    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8495c67bfc-xtngb"]
Sep 15 12:03:37.226106 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:37.226078    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:03:37.296175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:37.296127    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:37.390734 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:37.390696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:37.445780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:37.445693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:38.127005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:38.126967    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:38.233727 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:38.233650    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:38.335230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:38.335172    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:38.431729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:38.431639    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:41.292842 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:41.292803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:41.447998 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:41.447959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:44.498728 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:44.498692    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:44.562867 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:44.562831    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:44.624628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:44.624589    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:45.161648 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:45.161607    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:48.391177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:48.391138    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:48.493271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:48.493239    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:50.409957 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:50.409918    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b404c4b-5d10-407c-9193-4eb1fe994a97" containerID="d148b9d12f66905ae64a45b27622e4128c67b93a5a8b965df5060036cc04a23a" exitCode=0
Sep 15 12:03:50.410405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:50.409991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s5npn" event={"ID":"0b404c4b-5d10-407c-9193-4eb1fe994a97","Type":"ContainerDied","Data":"d148b9d12f66905ae64a45b27622e4128c67b93a5a8b965df5060036cc04a23a"}
Sep 15 12:03:50.410405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:50.410400    2583 scope.go:117] "RemoveContainer" containerID="d148b9d12f66905ae64a45b27622e4128c67b93a5a8b965df5060036cc04a23a"
Sep 15 12:03:51.278506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:51.278476    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847cffd544-fxqfj_ef818076-7989-462f-b9f6-92a7cc8352d8/router/0.log"
Sep 15 12:03:51.295690 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:51.295656    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zcjwh_81040b81-30da-48ef-963b-d42e89a16b58/serve-healthcheck-canary/0.log"
Sep 15 12:03:51.420882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:51.420842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s5npn" event={"ID":"0b404c4b-5d10-407c-9193-4eb1fe994a97","Type":"ContainerStarted","Data":"d02a8948d4292cd576ee170913a75002509116b5cf25f0ce9623197fa9b52cd3"}
Sep 15 12:03:52.115469 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:52.115411    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:03:52.214876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:03:52.214833    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:02.247241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.247159    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry" containerID="cri-o://4353ae2d072dc217c143b1e2a55f7780367ca00a982b8d3bb430401b3f592c2a" gracePeriod=30
Sep 15 12:04:02.467786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.467752    2583 generic.go:356] "Generic (PLEG): container finished" podID="23e5d6de-da28-4ac9-b31d-544108426598" containerID="4353ae2d072dc217c143b1e2a55f7780367ca00a982b8d3bb430401b3f592c2a" exitCode=0
Sep 15 12:04:02.467962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.467854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" event={"ID":"23e5d6de-da28-4ac9-b31d-544108426598","Type":"ContainerDied","Data":"4353ae2d072dc217c143b1e2a55f7780367ca00a982b8d3bb430401b3f592c2a"}
Sep 15 12:04:02.530253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.530219    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:04:02.677667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677635    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23e5d6de-da28-4ac9-b31d-544108426598-ca-trust-extracted\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-bound-sa-token\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677700    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677736    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5ht7\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-kube-api-access-p5ht7\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-installation-pull-secrets\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-image-registry-private-configuration\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-registry-certificates\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.677905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.677866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-trusted-ca\") pod \"23e5d6de-da28-4ac9-b31d-544108426598\" (UID: \"23e5d6de-da28-4ac9-b31d-544108426598\") "
Sep 15 12:04:02.678501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.678447    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:04:02.678630 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.678503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:04:02.681454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.681419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-kube-api-access-p5ht7" (OuterVolumeSpecName: "kube-api-access-p5ht7") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "kube-api-access-p5ht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:04:02.681711 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.681683    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:04:02.681941 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.681918    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:04:02.682036 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.681956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:04:02.682101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.682057    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:04:02.690222 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.690183    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e5d6de-da28-4ac9-b31d-544108426598-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "23e5d6de-da28-4ac9-b31d-544108426598" (UID: "23e5d6de-da28-4ac9-b31d-544108426598"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779494    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-installation-pull-secrets\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779539    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/23e5d6de-da28-4ac9-b31d-544108426598-image-registry-private-configuration\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779556    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-registry-certificates\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779574    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/23e5d6de-da28-4ac9-b31d-544108426598-trusted-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779591    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/23e5d6de-da28-4ac9-b31d-544108426598-ca-trust-extracted\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779606    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-bound-sa-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779621    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-registry-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:02.779715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:02.779635    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ht7\" (UniqueName: \"kubernetes.io/projected/23e5d6de-da28-4ac9-b31d-544108426598-kube-api-access-p5ht7\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:04:03.475639 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:03.475610    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb"
Sep 15 12:04:03.476074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:03.475609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8495c67bfc-xtngb" event={"ID":"23e5d6de-da28-4ac9-b31d-544108426598","Type":"ContainerDied","Data":"0a5a569b9c8079fef98a59e79b1812b9c376b343a93b171ff1c2300209c2f663"}
Sep 15 12:04:03.476074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:03.475746    2583 scope.go:117] "RemoveContainer" containerID="4353ae2d072dc217c143b1e2a55f7780367ca00a982b8d3bb430401b3f592c2a"
Sep 15 12:04:03.501753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:03.501724    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8495c67bfc-xtngb"]
Sep 15 12:04:03.504662 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:03.504628    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8495c67bfc-xtngb"]
Sep 15 12:04:03.806738 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:03.806656    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e5d6de-da28-4ac9-b31d-544108426598" path="/var/lib/kubelet/pods/23e5d6de-da28-4ac9-b31d-544108426598/volumes"
Sep 15 12:04:14.166116 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:14.166080    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:14.266754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:14.266712    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:14.331592 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:14.331559    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:20.545574 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:20.545541    2583 generic.go:356] "Generic (PLEG): container finished" podID="47fdb44b-602b-4b18-b9f7-e1a11db46547" containerID="2507d1d4a8dd64bf7099ae01d146b9ab1437378dcba731cade7dbec57ef092f8" exitCode=0
Sep 15 12:04:20.546090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:20.545619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj" event={"ID":"47fdb44b-602b-4b18-b9f7-e1a11db46547","Type":"ContainerDied","Data":"2507d1d4a8dd64bf7099ae01d146b9ab1437378dcba731cade7dbec57ef092f8"}
Sep 15 12:04:20.546090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:20.546071    2583 scope.go:117] "RemoveContainer" containerID="2507d1d4a8dd64bf7099ae01d146b9ab1437378dcba731cade7dbec57ef092f8"
Sep 15 12:04:20.547186 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:20.547164    2583 generic.go:356] "Generic (PLEG): container finished" podID="23b96874-988f-4db1-a224-bb08ea644d09" containerID="2dd05d9c71424dabb210d9270c71c28750fd5717627fff2f595b8c3b64f610ca" exitCode=0
Sep 15 12:04:20.547281 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:20.547208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r" event={"ID":"23b96874-988f-4db1-a224-bb08ea644d09","Type":"ContainerDied","Data":"2dd05d9c71424dabb210d9270c71c28750fd5717627fff2f595b8c3b64f610ca"}
Sep 15 12:04:20.547452 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:20.547440    2583 scope.go:117] "RemoveContainer" containerID="2dd05d9c71424dabb210d9270c71c28750fd5717627fff2f595b8c3b64f610ca"
Sep 15 12:04:21.552712 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:21.552671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ml4vj" event={"ID":"47fdb44b-602b-4b18-b9f7-e1a11db46547","Type":"ContainerStarted","Data":"45558ad498d72cff4e11582db89f3ff51b4ee7a20bb69ad267e4b7fe175829e7"}
Sep 15 12:04:21.554543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:21.554517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-f9g8r" event={"ID":"23b96874-988f-4db1-a224-bb08ea644d09","Type":"ContainerStarted","Data":"58a099ad3da0140dc8741a493b46611d75d62126604f92527f0a84c57117d4fe"}
Sep 15 12:04:43.728336 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:43.728294    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:43.829901 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:43.829868    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:52.707753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:52.707675    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:04:52.762237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:04:52.762198    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:03.599497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:03.599463    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:03.699514 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:03.699478    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:03.800737 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:03.800689    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:21.375860 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:21.375816    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:24.541421 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:24.541382    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:25.564478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:25.564438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:31.980662 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:31.980620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:58.961316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:58.961276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:05:59.024753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:05:59.024717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:10.955586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:10.955547    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:11.013561 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:11.013520    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:12.902573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:12.902540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:12.955160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:12.955120    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:22.127341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:22.127237    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:22.226471 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:22.226426    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:23.900720 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:23.900689    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:23.959375 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:23.959338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:24.020768 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:24.020734    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:06:59.702870 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:59.702840    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:06:59.708085 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:59.708012    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:06:59.715204 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:06:59.715182    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 12:07:04.392207 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:04.392171    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:05.775440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:05.775400    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:36.514180 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:36.514144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:37.924358 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:37.924319    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:40.156669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:40.156625    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:40.251296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:40.251262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:41.560661 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:41.560624    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:41.668181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:41.668144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:43.988976 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:43.988937    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:44.391259 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:44.391228    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:46.923567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:46.923532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:49.432512 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:49.432475    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:49.536262 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:49.536211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:51.545361 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:51.545322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:52.192308 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:52.192229    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:53.352589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:53.352550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:07:53.444673 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:07:53.444638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:00.865668 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:00.865629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:00.969146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:00.969106    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:03.286592 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:03.286557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:15.240731 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:15.240698    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:15.342830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:15.342802    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:15.858689 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:15.858652    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:16.117003 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.116887    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:16.194218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.194183    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:16.227909 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.227872    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"]
Sep 15 12:08:16.228345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.228325    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry"
Sep 15 12:08:16.228345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.228347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry"
Sep 15 12:08:16.228482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.228423    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e5d6de-da28-4ac9-b31d-544108426598" containerName="registry"
Sep 15 12:08:16.230326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.230304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.234554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.234534    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 12:08:16.234664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.234646    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-khcjr\""
Sep 15 12:08:16.234943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.234924    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 12:08:16.235230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.235213    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 12:08:16.235338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.235319    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 12:08:16.246674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.246649    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"]
Sep 15 12:08:16.326684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.326650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25757abf-ba99-4912-9785-d946796d63a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.326887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.326724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25757abf-ba99-4912-9785-d946796d63a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.326887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.326783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25757abf-ba99-4912-9785-d946796d63a4-manifests\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.326887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.326844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xx5v\" (UniqueName: \"kubernetes.io/projected/25757abf-ba99-4912-9785-d946796d63a4-kube-api-access-5xx5v\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.327053 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.326885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25757abf-ba99-4912-9785-d946796d63a4-tmp\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.338722 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.338690    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:16.428343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.428246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xx5v\" (UniqueName: \"kubernetes.io/projected/25757abf-ba99-4912-9785-d946796d63a4-kube-api-access-5xx5v\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.428343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.428301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25757abf-ba99-4912-9785-d946796d63a4-tmp\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.428343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.428336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25757abf-ba99-4912-9785-d946796d63a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.428620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.428398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25757abf-ba99-4912-9785-d946796d63a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.428620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.428426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25757abf-ba99-4912-9785-d946796d63a4-manifests\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.428824 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.428803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25757abf-ba99-4912-9785-d946796d63a4-manifests\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.430681 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.430657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25757abf-ba99-4912-9785-d946796d63a4-tmp\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.430815 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.430796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25757abf-ba99-4912-9785-d946796d63a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.430898 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.430880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25757abf-ba99-4912-9785-d946796d63a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.446873 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.446841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xx5v\" (UniqueName: \"kubernetes.io/projected/25757abf-ba99-4912-9785-d946796d63a4-kube-api-access-5xx5v\") pod \"opendatahub-operator-controller-manager-7b747b9585-p9wwk\" (UID: \"25757abf-ba99-4912-9785-d946796d63a4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.540559 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.540520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:16.825589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.825543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"]
Sep 15 12:08:16.830103 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:08:16.830079    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25757abf_ba99_4912_9785_d946796d63a4.slice/crio-a69dc37a184e487a6cb80a13a46d85b75595d384b4a5b19616be68d539dc1450 WatchSource:0}: Error finding container a69dc37a184e487a6cb80a13a46d85b75595d384b4a5b19616be68d539dc1450: Status 404 returned error can't find the container with id a69dc37a184e487a6cb80a13a46d85b75595d384b4a5b19616be68d539dc1450
Sep 15 12:08:16.831703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:16.831685    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:08:17.476897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:17.476845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk" event={"ID":"25757abf-ba99-4912-9785-d946796d63a4","Type":"ContainerStarted","Data":"a69dc37a184e487a6cb80a13a46d85b75595d384b4a5b19616be68d539dc1450"}
Sep 15 12:08:18.713894 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:18.713829    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:18.812332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:18.812288    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:20.492796 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:20.492754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk" event={"ID":"25757abf-ba99-4912-9785-d946796d63a4","Type":"ContainerStarted","Data":"6341ac2c9bdfd267823e43118b71f62b50ab9e2baff26097caa3aac1c13c04ec"}
Sep 15 12:08:21.498924 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:21.498886    2583 generic.go:356] "Generic (PLEG): container finished" podID="25757abf-ba99-4912-9785-d946796d63a4" containerID="6341ac2c9bdfd267823e43118b71f62b50ab9e2baff26097caa3aac1c13c04ec" exitCode=0
Sep 15 12:08:21.499345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:21.498977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk" event={"ID":"25757abf-ba99-4912-9785-d946796d63a4","Type":"ContainerDied","Data":"6341ac2c9bdfd267823e43118b71f62b50ab9e2baff26097caa3aac1c13c04ec"}
Sep 15 12:08:22.504959 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:22.504923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk" event={"ID":"25757abf-ba99-4912-9785-d946796d63a4","Type":"ContainerStarted","Data":"719fded5a2ce8206a359c94a3d49b92e2f792f5baa22ad3fad1701d5fc198974"}
Sep 15 12:08:22.505394 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:22.505051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:22.527830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:22.527780    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk" podStartSLOduration=2.279013158 podStartE2EDuration="6.527765016s" totalImagesPullingTime="3.207016691s" totalInitContainerRuntime="1.041735167s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:22.526292742 +0000 UTC m=+383.222852363" watchObservedRunningTime="2026-09-15 12:08:22.527765016 +0000 UTC m=+383.224324593"
Sep 15 12:08:23.517125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:23.517092    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:23.588263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:23.588231    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:26.950123 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:26.950081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:27.043834 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:27.043799    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:27.313644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:27.313555    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:27.418423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:27.418385    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:27.514777 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:27.514743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:28.115299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:28.115260    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:28.317534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:28.317497    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:33.512078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:33.512015    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b747b9585-p9wwk"
Sep 15 12:08:36.945903 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:36.945867    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:37.102356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.102325    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:37.377946 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.377859    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"]
Sep 15 12:08:37.385856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.385833    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.387799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.387771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbsjw\" (UniqueName: \"kubernetes.io/projected/a36df0a6-b85d-417f-97bf-e496036f2e1e-kube-api-access-nbsjw\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.387927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.387808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36df0a6-b85d-417f-97bf-e496036f2e1e-tmp\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.387927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.387891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36df0a6-b85d-417f-97bf-e496036f2e1e-tls-certs\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.390079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.390050    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:37.391002 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.390980    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 12:08:37.391147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.390983    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 12:08:37.396827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.396803    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"]
Sep 15 12:08:37.488921 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.488886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36df0a6-b85d-417f-97bf-e496036f2e1e-tls-certs\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.488921 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.488923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbsjw\" (UniqueName: \"kubernetes.io/projected/a36df0a6-b85d-417f-97bf-e496036f2e1e-kube-api-access-nbsjw\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.489144 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.488948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36df0a6-b85d-417f-97bf-e496036f2e1e-tmp\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.491282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.491248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36df0a6-b85d-417f-97bf-e496036f2e1e-tmp\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.491453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.491432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36df0a6-b85d-417f-97bf-e496036f2e1e-tls-certs\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.512658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.512629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbsjw\" (UniqueName: \"kubernetes.io/projected/a36df0a6-b85d-417f-97bf-e496036f2e1e-kube-api-access-nbsjw\") pod \"kube-auth-proxy-7cf78458b7-mk969\" (UID: \"a36df0a6-b85d-417f-97bf-e496036f2e1e\") " pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.695850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.695761    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"
Sep 15 12:08:37.987551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:37.987460    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cf78458b7-mk969"]
Sep 15 12:08:37.990388 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:08:37.990359    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36df0a6_b85d_417f_97bf_e496036f2e1e.slice/crio-75cd1c189d0f4bb6403fcf2a4c7f093669d37187d832e247f71005b8a24b8d17 WatchSource:0}: Error finding container 75cd1c189d0f4bb6403fcf2a4c7f093669d37187d832e247f71005b8a24b8d17: Status 404 returned error can't find the container with id 75cd1c189d0f4bb6403fcf2a4c7f093669d37187d832e247f71005b8a24b8d17
Sep 15 12:08:38.575944 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:38.575906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969" event={"ID":"a36df0a6-b85d-417f-97bf-e496036f2e1e","Type":"ContainerStarted","Data":"75cd1c189d0f4bb6403fcf2a4c7f093669d37187d832e247f71005b8a24b8d17"}
Sep 15 12:08:39.415532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.415465    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:39.466652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.466600    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"]
Sep 15 12:08:39.469260 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.469232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.471629 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.471602    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 12:08:39.471808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.471786    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 12:08:39.471941 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.471930    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zp672\""
Sep 15 12:08:39.472064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.472038    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 12:08:39.472197 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.472182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 12:08:39.492004 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.491928    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"]
Sep 15 12:08:39.503081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.503244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.503244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.503244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0e7-4a5b-432a-989d-214db28125a3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.503415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkhk\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-kube-api-access-szkhk\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.503415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.503415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.503328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.505848 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.505803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:39.582576 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.582539    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0e7-4a5b-432a-989d-214db28125a3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkhk\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-kube-api-access-szkhk\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.604778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.604581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.609453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.606362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.609453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.607861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.609453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.609122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.609855 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.609495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.609855 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.609630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0e7-4a5b-432a-989d-214db28125a3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.614607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.613685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkhk\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-kube-api-access-szkhk\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.616687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.616662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-8bfmb\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:39.783638 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:39.783555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:40.494631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:40.494591    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:40.790674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:40.790648    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"]
Sep 15 12:08:41.169342 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:08:41.169305    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e1e0e7_4a5b_432a_989d_214db28125a3.slice/crio-0dd31121db6768bb59883cdc2cd151e4d0fc67ef4dd601c7d63c712e05242c6d WatchSource:0}: Error finding container 0dd31121db6768bb59883cdc2cd151e4d0fc67ef4dd601c7d63c712e05242c6d: Status 404 returned error can't find the container with id 0dd31121db6768bb59883cdc2cd151e4d0fc67ef4dd601c7d63c712e05242c6d
Sep 15 12:08:41.591867 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:41.591824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" event={"ID":"c8e1e0e7-4a5b-432a-989d-214db28125a3","Type":"ContainerStarted","Data":"0dd31121db6768bb59883cdc2cd151e4d0fc67ef4dd601c7d63c712e05242c6d"}
Sep 15 12:08:41.593311 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:41.593285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969" event={"ID":"a36df0a6-b85d-417f-97bf-e496036f2e1e","Type":"ContainerStarted","Data":"5a7d5e940a767cb8e8588caf9d959eaa7a3231beba68803b4e014b1286017adc"}
Sep 15 12:08:41.609171 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:41.609118    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cf78458b7-mk969" podStartSLOduration=1.372688296 podStartE2EDuration="4.609102732s" totalImagesPullingTime="3.236414436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:41.608314386 +0000 UTC m=+402.304873966" watchObservedRunningTime="2026-09-15 12:08:41.609102732 +0000 UTC m=+402.305662332"
Sep 15 12:08:43.574098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:43.574062    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:08:43.574363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:43.574134    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:08:44.294076 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.294019    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:44.393952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.393912    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:44.611083 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.611046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" event={"ID":"c8e1e0e7-4a5b-432a-989d-214db28125a3","Type":"ContainerStarted","Data":"cf88739315db98e8910c819fe85cf56fdc84966b77565f30d00bf3be1555503a"}
Sep 15 12:08:44.611544 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.611241    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:44.612761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.612729    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-8bfmb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:08:44.612876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.612791    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:08:44.634735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.634701    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:44.639776 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:44.639728    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" podStartSLOduration=3.237437804 podStartE2EDuration="5.63971132s" totalImagesPullingTime="2.402273516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:44.638395401 +0000 UTC m=+405.334954991" watchObservedRunningTime="2026-09-15 12:08:44.63971132 +0000 UTC m=+405.336270897"
Sep 15 12:08:45.617089 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:45.617059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:08:46.453464 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:46.453426    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:46.554948 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:46.554915    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:48.283556 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:48.283517    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:48.384566 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:48.384533    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:48.488450 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:48.488415    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:48.588178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:48.588140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:48.866013 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:48.865921    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:48.982092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:48.982059    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:58.949836 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:58.949797    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:59.068482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.068450    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"]
Sep 15 12:08:59.071847 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.071818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.076741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.076720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9fbrv\""
Sep 15 12:08:59.077260 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.077240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 12:08:59.103195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.103165    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"]
Sep 15 12:08:59.126733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.126695    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:59.172821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.172779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpvw\" (UniqueName: \"kubernetes.io/projected/cc2173ee-9149-4649-810b-440af589aa66-kube-api-access-gwpvw\") pod \"kserve-controller-manager-d6f4b65b8-45ggs\" (UID: \"cc2173ee-9149-4649-810b-440af589aa66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.172997 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.172842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc2173ee-9149-4649-810b-440af589aa66-cert\") pod \"kserve-controller-manager-d6f4b65b8-45ggs\" (UID: \"cc2173ee-9149-4649-810b-440af589aa66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.234169 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.234067    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:08:59.273506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.273472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpvw\" (UniqueName: \"kubernetes.io/projected/cc2173ee-9149-4649-810b-440af589aa66-kube-api-access-gwpvw\") pod \"kserve-controller-manager-d6f4b65b8-45ggs\" (UID: \"cc2173ee-9149-4649-810b-440af589aa66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.273699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.273530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc2173ee-9149-4649-810b-440af589aa66-cert\") pod \"kserve-controller-manager-d6f4b65b8-45ggs\" (UID: \"cc2173ee-9149-4649-810b-440af589aa66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.276131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.276107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc2173ee-9149-4649-810b-440af589aa66-cert\") pod \"kserve-controller-manager-d6f4b65b8-45ggs\" (UID: \"cc2173ee-9149-4649-810b-440af589aa66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.281449 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.281423    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpvw\" (UniqueName: \"kubernetes.io/projected/cc2173ee-9149-4649-810b-440af589aa66-kube-api-access-gwpvw\") pod \"kserve-controller-manager-d6f4b65b8-45ggs\" (UID: \"cc2173ee-9149-4649-810b-440af589aa66\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.381326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.381274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:08:59.655567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.655532    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"]
Sep 15 12:08:59.658569 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:08:59.658537    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2173ee_9149_4649_810b_440af589aa66.slice/crio-8705ec65c019dc9ecb2ae943e6c0fb49442d23002bded0b9275eca3216f33121 WatchSource:0}: Error finding container 8705ec65c019dc9ecb2ae943e6c0fb49442d23002bded0b9275eca3216f33121: Status 404 returned error can't find the container with id 8705ec65c019dc9ecb2ae943e6c0fb49442d23002bded0b9275eca3216f33121
Sep 15 12:08:59.677952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:08:59.677924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs" event={"ID":"cc2173ee-9149-4649-810b-440af589aa66","Type":"ContainerStarted","Data":"8705ec65c019dc9ecb2ae943e6c0fb49442d23002bded0b9275eca3216f33121"}
Sep 15 12:09:00.296182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.296141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:00.428406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.428375    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:00.509261 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.509226    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"]
Sep 15 12:09:00.509499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.509473    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerName="discovery" containerID="cri-o://cf88739315db98e8910c819fe85cf56fdc84966b77565f30d00bf3be1555503a" gracePeriod=30
Sep 15 12:09:00.529694 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.529655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:00.617000 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.616960    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-8bfmb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/ready\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Sep 15 12:09:00.617188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.617046    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.29:8080/ready\": dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 15 12:09:00.686940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.686905    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerID="cf88739315db98e8910c819fe85cf56fdc84966b77565f30d00bf3be1555503a" exitCode=0
Sep 15 12:09:00.687091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.686966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" event={"ID":"c8e1e0e7-4a5b-432a-989d-214db28125a3","Type":"ContainerDied","Data":"cf88739315db98e8910c819fe85cf56fdc84966b77565f30d00bf3be1555503a"}
Sep 15 12:09:00.783552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.783524    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:09:00.888964 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.888872    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szkhk\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-kube-api-access-szkhk\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.888964 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.888935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-token\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.889248 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.888979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-kubeconfig\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.889248 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.888999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-cacerts\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.889248 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.889109    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0e7-4a5b-432a-989d-214db28125a3-local-certs\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.889248 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.889169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-ca-configmap\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.889248 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.889234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-dns-cert\") pod \"c8e1e0e7-4a5b-432a-989d-214db28125a3\" (UID: \"c8e1e0e7-4a5b-432a-989d-214db28125a3\") "
Sep 15 12:09:00.889672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.889635    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:09:00.891420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.891365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:09:00.891533 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.891465    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-kube-api-access-szkhk" (OuterVolumeSpecName: "kube-api-access-szkhk") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "kube-api-access-szkhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:09:00.891611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.891593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:09:00.891692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.891612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-cacerts" (OuterVolumeSpecName: "cacerts") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:09:00.891765 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.891744    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:09:00.891821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.891777    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e1e0e7-4a5b-432a-989d-214db28125a3-local-certs" (OuterVolumeSpecName: "local-certs") pod "c8e1e0e7-4a5b-432a-989d-214db28125a3" (UID: "c8e1e0e7-4a5b-432a-989d-214db28125a3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:09:00.990578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990540    2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e1e0e7-4a5b-432a-989d-214db28125a3-local-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:00.990578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990573    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-ca-configmap\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:00.990578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990583    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-csr-dns-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:00.990806 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990592    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szkhk\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-kube-api-access-szkhk\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:00.990806 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990601    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:00.990806 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990611    2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-istio-kubeconfig\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:00.990806 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:00.990618    2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8e1e0e7-4a5b-432a-989d-214db28125a3-cacerts\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:09:01.691980 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:01.691950    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"
Sep 15 12:09:01.691980 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:01.691978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb" event={"ID":"c8e1e0e7-4a5b-432a-989d-214db28125a3","Type":"ContainerDied","Data":"0dd31121db6768bb59883cdc2cd151e4d0fc67ef4dd601c7d63c712e05242c6d"}
Sep 15 12:09:01.692476 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:01.692041    2583 scope.go:117] "RemoveContainer" containerID="cf88739315db98e8910c819fe85cf56fdc84966b77565f30d00bf3be1555503a"
Sep 15 12:09:01.715341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:01.715309    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"]
Sep 15 12:09:01.723793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:01.723753    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-8bfmb"]
Sep 15 12:09:01.805430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:01.805397    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" path="/var/lib/kubelet/pods/c8e1e0e7-4a5b-432a-989d-214db28125a3/volumes"
Sep 15 12:09:04.709871 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:04.709836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs" event={"ID":"cc2173ee-9149-4649-810b-440af589aa66","Type":"ContainerStarted","Data":"1799b18db46190512a79053f754732d59254d856b7999128b2eaf4ca271d585c"}
Sep 15 12:09:04.710424 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:04.709942    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:09:04.730560 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:04.730514    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs" podStartSLOduration=1.588092433 podStartE2EDuration="5.730497528s" totalImagesPullingTime="4.142405095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:09:04.729356446 +0000 UTC m=+425.425916059" watchObservedRunningTime="2026-09-15 12:09:04.730497528 +0000 UTC m=+425.427057107"
Sep 15 12:09:35.719489 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:35.719458    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-45ggs"
Sep 15 12:09:40.610279 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:40.610237    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:40.711098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:40.711054    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:47.309588 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:47.309540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:48.802042 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:48.801992    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:49.278085 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:49.278047    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:49.472550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:49.472512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:49.698372 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:49.698339    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:09:49.879006 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:09:49.878955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:05.441444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:05.441396    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:07.596511 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.596471    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"]
Sep 15 12:10:07.596984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.596823    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerName="discovery"
Sep 15 12:10:07.596984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.596834    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerName="discovery"
Sep 15 12:10:07.596984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.596898    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e1e0e7-4a5b-432a-989d-214db28125a3" containerName="discovery"
Sep 15 12:10:07.599142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.599123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:07.611156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.611132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 12:10:07.611274 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.611131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 12:10:07.612127 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.612105    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5cjwf\""
Sep 15 12:10:07.612229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.612168    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 12:10:07.648761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.648732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"]
Sep 15 12:10:07.749088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.749054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjcl\" (UniqueName: \"kubernetes.io/projected/e6de30c0-901b-4880-83a9-70ca1ad48718-kube-api-access-bkjcl\") pod \"dns-operator-controller-manager-7b84cd7c9c-b88lm\" (UID: \"e6de30c0-901b-4880-83a9-70ca1ad48718\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:07.850320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.850236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjcl\" (UniqueName: \"kubernetes.io/projected/e6de30c0-901b-4880-83a9-70ca1ad48718-kube-api-access-bkjcl\") pod \"dns-operator-controller-manager-7b84cd7c9c-b88lm\" (UID: \"e6de30c0-901b-4880-83a9-70ca1ad48718\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:07.872404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.872372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjcl\" (UniqueName: \"kubernetes.io/projected/e6de30c0-901b-4880-83a9-70ca1ad48718-kube-api-access-bkjcl\") pod \"dns-operator-controller-manager-7b84cd7c9c-b88lm\" (UID: \"e6de30c0-901b-4880-83a9-70ca1ad48718\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:07.908913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.908863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:07.982483 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:07.982439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:08.196076 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:08.195905    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"]
Sep 15 12:10:08.198966 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:08.198940    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6de30c0_901b_4880_83a9_70ca1ad48718.slice/crio-ce0c23e9277b1b73a4ecc3aa474a8d04615a43ffbe3b85de7411163c794470b1 WatchSource:0}: Error finding container ce0c23e9277b1b73a4ecc3aa474a8d04615a43ffbe3b85de7411163c794470b1: Status 404 returned error can't find the container with id ce0c23e9277b1b73a4ecc3aa474a8d04615a43ffbe3b85de7411163c794470b1
Sep 15 12:10:08.307517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:08.307481    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:08.990303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:08.990266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm" event={"ID":"e6de30c0-901b-4880-83a9-70ca1ad48718","Type":"ContainerStarted","Data":"ce0c23e9277b1b73a4ecc3aa474a8d04615a43ffbe3b85de7411163c794470b1"}
Sep 15 12:10:09.436309 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:09.436273    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:09.546555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:09.546517    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:10.494518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.494476    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-lmbsz"]
Sep 15 12:10:10.496746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.496724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:10.502316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.502293    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4vwhl\""
Sep 15 12:10:10.520388 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.520360    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-lmbsz"]
Sep 15 12:10:10.574436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.574404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4462\" (UniqueName: \"kubernetes.io/projected/569e7a31-2a8e-4364-b182-146adc5d1c3f-kube-api-access-c4462\") pod \"authorino-operator-76b478bd78-lmbsz\" (UID: \"569e7a31-2a8e-4364-b182-146adc5d1c3f\") " pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:10.675590 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.675554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4462\" (UniqueName: \"kubernetes.io/projected/569e7a31-2a8e-4364-b182-146adc5d1c3f-kube-api-access-c4462\") pod \"authorino-operator-76b478bd78-lmbsz\" (UID: \"569e7a31-2a8e-4364-b182-146adc5d1c3f\") " pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:10.690685 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.690650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4462\" (UniqueName: \"kubernetes.io/projected/569e7a31-2a8e-4364-b182-146adc5d1c3f-kube-api-access-c4462\") pod \"authorino-operator-76b478bd78-lmbsz\" (UID: \"569e7a31-2a8e-4364-b182-146adc5d1c3f\") " pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:10.790138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.790053    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:10.808972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.808927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:10.932881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:10.932843    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:11.634810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:11.634780    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:11.851655 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:11.851572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-lmbsz"]
Sep 15 12:10:11.854959 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:11.854921    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569e7a31_2a8e_4364_b182_146adc5d1c3f.slice/crio-4606b2204fb795b64a8bdfb0b4fc1766b0bfd9c404a6b50a276015c6f44ef349 WatchSource:0}: Error finding container 4606b2204fb795b64a8bdfb0b4fc1766b0bfd9c404a6b50a276015c6f44ef349: Status 404 returned error can't find the container with id 4606b2204fb795b64a8bdfb0b4fc1766b0bfd9c404a6b50a276015c6f44ef349
Sep 15 12:10:12.007035 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.006999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm" event={"ID":"e6de30c0-901b-4880-83a9-70ca1ad48718","Type":"ContainerStarted","Data":"ccc2a25e1ca47ce1cd46e0bf276655a1f0a12a73411171a0cf9c19814806d4f0"}
Sep 15 12:10:12.007243 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.007088    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:12.008499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.008473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz" event={"ID":"569e7a31-2a8e-4364-b182-146adc5d1c3f","Type":"ContainerStarted","Data":"4606b2204fb795b64a8bdfb0b4fc1766b0bfd9c404a6b50a276015c6f44ef349"}
Sep 15 12:10:12.110881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.110802    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:12.426214 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.426089    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm" podStartSLOduration=2.024424055 podStartE2EDuration="5.426065074s" totalImagesPullingTime="3.401641019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:12.048790036 +0000 UTC m=+492.745349614" watchObservedRunningTime="2026-09-15 12:10:12.426065074 +0000 UTC m=+493.122624654"
Sep 15 12:10:12.427247 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.427224    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"]
Sep 15 12:10:12.429360 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.429343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:12.433220 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.433198    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nqvwl\""
Sep 15 12:10:12.443763 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.443740    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"]
Sep 15 12:10:12.593009 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.592970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8l2\" (UniqueName: \"kubernetes.io/projected/392a6486-e73e-4432-97af-7f613fb3818c-kube-api-access-sj8l2\") pod \"limitador-operator-controller-manager-74495c69f8-jwwh2\" (UID: \"392a6486-e73e-4432-97af-7f613fb3818c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:12.694566 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.694477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8l2\" (UniqueName: \"kubernetes.io/projected/392a6486-e73e-4432-97af-7f613fb3818c-kube-api-access-sj8l2\") pod \"limitador-operator-controller-manager-74495c69f8-jwwh2\" (UID: \"392a6486-e73e-4432-97af-7f613fb3818c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:12.704268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.704238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8l2\" (UniqueName: \"kubernetes.io/projected/392a6486-e73e-4432-97af-7f613fb3818c-kube-api-access-sj8l2\") pod \"limitador-operator-controller-manager-74495c69f8-jwwh2\" (UID: \"392a6486-e73e-4432-97af-7f613fb3818c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:12.740093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.740054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:12.858684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:12.858635    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:13.027846 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:13.027817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"]
Sep 15 12:10:13.100620 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:13.100589    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392a6486_e73e_4432_97af_7f613fb3818c.slice/crio-7bb96cb2a68d76612b0aa3b4eb11905fe0b3fe8b650ac1094198304839031317 WatchSource:0}: Error finding container 7bb96cb2a68d76612b0aa3b4eb11905fe0b3fe8b650ac1094198304839031317: Status 404 returned error can't find the container with id 7bb96cb2a68d76612b0aa3b4eb11905fe0b3fe8b650ac1094198304839031317
Sep 15 12:10:13.218350 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:13.218227    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:13.451569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:13.451531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:13.569847 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:13.569814    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:13.667893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:13.667860    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:13.774425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:13.774336    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:14.020902 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:14.020869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz" event={"ID":"569e7a31-2a8e-4364-b182-146adc5d1c3f","Type":"ContainerStarted","Data":"c849f7ac9d1bc2dfeca31a6c91b8bb1160a8a43a2488e0291c9e4034b2143289"}
Sep 15 12:10:14.021107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:14.021001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:14.022339 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:14.022316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2" event={"ID":"392a6486-e73e-4432-97af-7f613fb3818c","Type":"ContainerStarted","Data":"7bb96cb2a68d76612b0aa3b4eb11905fe0b3fe8b650ac1094198304839031317"}
Sep 15 12:10:14.052758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:14.052667    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz" podStartSLOduration=2.780998479 podStartE2EDuration="4.052652077s" totalImagesPullingTime="1.271653598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:14.05135034 +0000 UTC m=+494.747909942" watchObservedRunningTime="2026-09-15 12:10:14.052652077 +0000 UTC m=+494.749211654"
Sep 15 12:10:16.220298 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.220261    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"]
Sep 15 12:10:16.222709 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.222687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.226655 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.226636    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-49cwg\""
Sep 15 12:10:16.241555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.241524    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"]
Sep 15 12:10:16.325735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.325691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995cdf1d-5fed-41e4-9a64-82a32b212e3b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.325926 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.325762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7fl\" (UniqueName: \"kubernetes.io/projected/995cdf1d-5fed-41e4-9a64-82a32b212e3b-kube-api-access-cx7fl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.427167 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.427127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995cdf1d-5fed-41e4-9a64-82a32b212e3b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.427315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.427218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7fl\" (UniqueName: \"kubernetes.io/projected/995cdf1d-5fed-41e4-9a64-82a32b212e3b-kube-api-access-cx7fl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.427531 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.427512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995cdf1d-5fed-41e4-9a64-82a32b212e3b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.436546 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.436516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7fl\" (UniqueName: \"kubernetes.io/projected/995cdf1d-5fed-41e4-9a64-82a32b212e3b-kube-api-access-cx7fl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.532907 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.532824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:16.814340 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:16.814309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"]
Sep 15 12:10:16.817202 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:16.817178    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995cdf1d_5fed_41e4_9a64_82a32b212e3b.slice/crio-037948019a5ef6effaa5dd8cc586ff2edab60e20615c62bc3a613c6ad17240fc WatchSource:0}: Error finding container 037948019a5ef6effaa5dd8cc586ff2edab60e20615c62bc3a613c6ad17240fc: Status 404 returned error can't find the container with id 037948019a5ef6effaa5dd8cc586ff2edab60e20615c62bc3a613c6ad17240fc
Sep 15 12:10:17.038251 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:17.038147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c" event={"ID":"995cdf1d-5fed-41e4-9a64-82a32b212e3b","Type":"ContainerStarted","Data":"037948019a5ef6effaa5dd8cc586ff2edab60e20615c62bc3a613c6ad17240fc"}
Sep 15 12:10:18.745477 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:18.745394    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:18.898866 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:18.898823    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:21.061155 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:21.061082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c" event={"ID":"995cdf1d-5fed-41e4-9a64-82a32b212e3b","Type":"ContainerStarted","Data":"882a3efcf9218e1c68a414705ad2eca0c542d6b531614b217d60a3c62067e5be"}
Sep 15 12:10:21.061532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:21.061222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:21.111692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:21.111638    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c" podStartSLOduration=1.605432898 podStartE2EDuration="5.111621751s" totalImagesPullingTime="3.506188853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:21.109829158 +0000 UTC m=+501.806388736" watchObservedRunningTime="2026-09-15 12:10:21.111621751 +0000 UTC m=+501.808181328"
Sep 15 12:10:22.970586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:22.970552    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:23.015609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:23.015581    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-b88lm"
Sep 15 12:10:23.096548 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:23.096515    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:23.312327 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:23.312229    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:23.418832 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:23.418788    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:25.029721 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:25.029693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-lmbsz"
Sep 15 12:10:26.335686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:26.335662    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:26.469283 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:26.469243    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:27.091423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:27.091387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2" event={"ID":"392a6486-e73e-4432-97af-7f613fb3818c","Type":"ContainerStarted","Data":"81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e"}
Sep 15 12:10:27.091594 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:27.091564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:27.122969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:27.122918    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2" podStartSLOduration=1.89456644 podStartE2EDuration="15.122901443s" totalImagesPullingTime="13.228335003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:27.120212362 +0000 UTC m=+507.816771962" watchObservedRunningTime="2026-09-15 12:10:27.122901443 +0000 UTC m=+507.819461020"
Sep 15 12:10:31.290365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:31.290324    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:31.421606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:31.421574    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:31.563864 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:31.563770    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:31.860299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:31.860257    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:32.068305 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:32.068271    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:34.964753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:34.964713    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"]
Sep 15 12:10:34.965198 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:34.964901    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" containerName="manager" containerID="cri-o://882a3efcf9218e1c68a414705ad2eca0c542d6b531614b217d60a3c62067e5be" gracePeriod=2
Sep 15 12:10:35.010568 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.010529    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"]
Sep 15 12:10:35.049236 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.049200    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"]
Sep 15 12:10:35.049432 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.049405    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2" podUID="392a6486-e73e-4432-97af-7f613fb3818c" containerName="manager" containerID="cri-o://81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e" gracePeriod=2
Sep 15 12:10:35.051375 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.051348    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:35.094895 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.094866    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"]
Sep 15 12:10:35.106416 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.106383    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:35.109964 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.109941    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwwh2\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="392a6486-e73e-4432-97af-7f613fb3818c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:35.127153 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.127119    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwwh2\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="392a6486-e73e-4432-97af-7f613fb3818c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:35.129816 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.129787    2583 generic.go:356] "Generic (PLEG): container finished" podID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" containerID="882a3efcf9218e1c68a414705ad2eca0c542d6b531614b217d60a3c62067e5be" exitCode=0
Sep 15 12:10:35.134917 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.134889    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:35.161525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161494    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"]
Sep 15 12:10:35.161858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161843    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="392a6486-e73e-4432-97af-7f613fb3818c" containerName="manager"
Sep 15 12:10:35.161858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161859    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="392a6486-e73e-4432-97af-7f613fb3818c" containerName="manager"
Sep 15 12:10:35.162005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161872    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" containerName="manager"
Sep 15 12:10:35.162005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161878    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" containerName="manager"
Sep 15 12:10:35.162005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161957    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" containerName="manager"
Sep 15 12:10:35.162005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.161965    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="392a6486-e73e-4432-97af-7f613fb3818c" containerName="manager"
Sep 15 12:10:35.163935 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.163910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.170380 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.170329    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:35.178931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.178892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5429555d-7a61-4eae-acc1-0c1a4a58ead5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4vxz9\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.179068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.178950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbrc\" (UniqueName: \"kubernetes.io/projected/5429555d-7a61-4eae-acc1-0c1a4a58ead5-kube-api-access-phbrc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4vxz9\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.197141 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.197097    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"]
Sep 15 12:10:35.199210 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.199187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:35.220692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.220621    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"]
Sep 15 12:10:35.280328 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.280297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkb8g\" (UniqueName: \"kubernetes.io/projected/50fd61a8-ba57-4d44-9265-0da863990d31-kube-api-access-xkb8g\") pod \"limitador-operator-controller-manager-74495c69f8-cs744\" (UID: \"50fd61a8-ba57-4d44-9265-0da863990d31\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:35.280457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.280343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5429555d-7a61-4eae-acc1-0c1a4a58ead5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4vxz9\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.280457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.280403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbrc\" (UniqueName: \"kubernetes.io/projected/5429555d-7a61-4eae-acc1-0c1a4a58ead5-kube-api-access-phbrc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4vxz9\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.280643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.280624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5429555d-7a61-4eae-acc1-0c1a4a58ead5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4vxz9\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.302947 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.302922    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:35.305415 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.305396    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:35.311819 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.311798    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"]
Sep 15 12:10:35.321396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.321373    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbrc\" (UniqueName: \"kubernetes.io/projected/5429555d-7a61-4eae-acc1-0c1a4a58ead5-kube-api-access-phbrc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4vxz9\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.381358 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.381322    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj8l2\" (UniqueName: \"kubernetes.io/projected/392a6486-e73e-4432-97af-7f613fb3818c-kube-api-access-sj8l2\") pod \"392a6486-e73e-4432-97af-7f613fb3818c\" (UID: \"392a6486-e73e-4432-97af-7f613fb3818c\") "
Sep 15 12:10:35.381548 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.381393    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995cdf1d-5fed-41e4-9a64-82a32b212e3b-extensions-socket-volume\") pod \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") "
Sep 15 12:10:35.381548 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.381417    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx7fl\" (UniqueName: \"kubernetes.io/projected/995cdf1d-5fed-41e4-9a64-82a32b212e3b-kube-api-access-cx7fl\") pod \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\" (UID: \"995cdf1d-5fed-41e4-9a64-82a32b212e3b\") "
Sep 15 12:10:35.381656 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.381564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkb8g\" (UniqueName: \"kubernetes.io/projected/50fd61a8-ba57-4d44-9265-0da863990d31-kube-api-access-xkb8g\") pod \"limitador-operator-controller-manager-74495c69f8-cs744\" (UID: \"50fd61a8-ba57-4d44-9265-0da863990d31\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:35.381943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.381914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995cdf1d-5fed-41e4-9a64-82a32b212e3b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "995cdf1d-5fed-41e4-9a64-82a32b212e3b" (UID: "995cdf1d-5fed-41e4-9a64-82a32b212e3b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:35.383651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.383621    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995cdf1d-5fed-41e4-9a64-82a32b212e3b-kube-api-access-cx7fl" (OuterVolumeSpecName: "kube-api-access-cx7fl") pod "995cdf1d-5fed-41e4-9a64-82a32b212e3b" (UID: "995cdf1d-5fed-41e4-9a64-82a32b212e3b"). InnerVolumeSpecName "kube-api-access-cx7fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:35.383749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.383671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392a6486-e73e-4432-97af-7f613fb3818c-kube-api-access-sj8l2" (OuterVolumeSpecName: "kube-api-access-sj8l2") pod "392a6486-e73e-4432-97af-7f613fb3818c" (UID: "392a6486-e73e-4432-97af-7f613fb3818c"). InnerVolumeSpecName "kube-api-access-sj8l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:35.403075 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.403042    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwwh2\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="392a6486-e73e-4432-97af-7f613fb3818c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:35.427965 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.427927    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwwh2\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="392a6486-e73e-4432-97af-7f613fb3818c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:35.429798 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.429778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkb8g\" (UniqueName: \"kubernetes.io/projected/50fd61a8-ba57-4d44-9265-0da863990d31-kube-api-access-xkb8g\") pod \"limitador-operator-controller-manager-74495c69f8-cs744\" (UID: \"50fd61a8-ba57-4d44-9265-0da863990d31\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:35.474791 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.474724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:35.482940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.482921    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8l2\" (UniqueName: \"kubernetes.io/projected/392a6486-e73e-4432-97af-7f613fb3818c-kube-api-access-sj8l2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:10:35.482999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.482944    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995cdf1d-5fed-41e4-9a64-82a32b212e3b-extensions-socket-volume\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:10:35.482999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.482955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7fl\" (UniqueName: \"kubernetes.io/projected/995cdf1d-5fed-41e4-9a64-82a32b212e3b-kube-api-access-cx7fl\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:10:35.487266 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:35.487237    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ddb4c\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:35.511695 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.511449    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:35.795977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.795893    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"]
Sep 15 12:10:35.799837 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:35.799807    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429555d_7a61_4eae_acc1_0c1a4a58ead5.slice/crio-c6ca1670a3fdef46571924d348a3708f6ac47e0808658b5944f808f9f7466424 WatchSource:0}: Error finding container c6ca1670a3fdef46571924d348a3708f6ac47e0808658b5944f808f9f7466424: Status 404 returned error can't find the container with id c6ca1670a3fdef46571924d348a3708f6ac47e0808658b5944f808f9f7466424
Sep 15 12:10:35.806033 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.805998    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="392a6486-e73e-4432-97af-7f613fb3818c" path="/var/lib/kubelet/pods/392a6486-e73e-4432-97af-7f613fb3818c/volumes"
Sep 15 12:10:35.806481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.806462    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="995cdf1d-5fed-41e4-9a64-82a32b212e3b" path="/var/lib/kubelet/pods/995cdf1d-5fed-41e4-9a64-82a32b212e3b/volumes"
Sep 15 12:10:35.850932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:35.850908    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"]
Sep 15 12:10:35.853818 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:35.853794    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fd61a8_ba57_4d44_9265_0da863990d31.slice/crio-bd374df7e8df74b0ef660f53aa0348f5fe400d5a446fa54056e9eb44088df999 WatchSource:0}: Error finding container bd374df7e8df74b0ef660f53aa0348f5fe400d5a446fa54056e9eb44088df999: Status 404 returned error can't find the container with id bd374df7e8df74b0ef660f53aa0348f5fe400d5a446fa54056e9eb44088df999
Sep 15 12:10:36.136460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.136427    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ddb4c"
Sep 15 12:10:36.136460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.136448    2583 scope.go:117] "RemoveContainer" containerID="882a3efcf9218e1c68a414705ad2eca0c542d6b531614b217d60a3c62067e5be"
Sep 15 12:10:36.137999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.137976    2583 generic.go:356] "Generic (PLEG): container finished" podID="392a6486-e73e-4432-97af-7f613fb3818c" containerID="81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e" exitCode=0
Sep 15 12:10:36.138100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.138054    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwwh2"
Sep 15 12:10:36.140195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.140151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9" event={"ID":"5429555d-7a61-4eae-acc1-0c1a4a58ead5","Type":"ContainerStarted","Data":"ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60"}
Sep 15 12:10:36.140318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.140229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9" event={"ID":"5429555d-7a61-4eae-acc1-0c1a4a58ead5","Type":"ContainerStarted","Data":"c6ca1670a3fdef46571924d348a3708f6ac47e0808658b5944f808f9f7466424"}
Sep 15 12:10:36.140318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.140285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:36.141998 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.141973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744" event={"ID":"50fd61a8-ba57-4d44-9265-0da863990d31","Type":"ContainerStarted","Data":"5114fe4541ef8e64432ce7ead7d111d6e16b2e4b6a533aa2aaf5c233e0fd3644"}
Sep 15 12:10:36.142092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.142045    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:36.142092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.142072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744" event={"ID":"50fd61a8-ba57-4d44-9265-0da863990d31","Type":"ContainerStarted","Data":"bd374df7e8df74b0ef660f53aa0348f5fe400d5a446fa54056e9eb44088df999"}
Sep 15 12:10:36.145780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.145764    2583 scope.go:117] "RemoveContainer" containerID="81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e"
Sep 15 12:10:36.153265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.153249    2583 scope.go:117] "RemoveContainer" containerID="81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e"
Sep 15 12:10:36.153533 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:36.153516    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e\": container with ID starting with 81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e not found: ID does not exist" containerID="81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e"
Sep 15 12:10:36.153584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.153555    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e"} err="failed to get container status \"81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e\": rpc error: code = NotFound desc = could not find container \"81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e\": container with ID starting with 81facaaa8661a32518500ee14a783901288364107c94832a89c138eb931bee7e not found: ID does not exist"
Sep 15 12:10:36.192482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.192434    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9" podStartSLOduration=1.192416605 podStartE2EDuration="1.192416605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:36.192403219 +0000 UTC m=+516.888962808" watchObservedRunningTime="2026-09-15 12:10:36.192416605 +0000 UTC m=+516.888976183"
Sep 15 12:10:36.246559 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:36.246498    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744" podStartSLOduration=1.246481965 podStartE2EDuration="1.246481965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:36.244111492 +0000 UTC m=+516.940671081" watchObservedRunningTime="2026-09-15 12:10:36.246481965 +0000 UTC m=+516.943041542"
Sep 15 12:10:38.700501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:38.700466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:38.986346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:38.986252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:39.246343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.246261    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"]
Sep 15 12:10:39.249151 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.249130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.278058 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.278012    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"]
Sep 15 12:10:39.313926 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.313888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2341045-6356-4333-b9fb-e3a7adfdbc14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-88bt7\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.314146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.314014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q97z\" (UniqueName: \"kubernetes.io/projected/d2341045-6356-4333-b9fb-e3a7adfdbc14-kube-api-access-8q97z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-88bt7\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.414583 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.414542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q97z\" (UniqueName: \"kubernetes.io/projected/d2341045-6356-4333-b9fb-e3a7adfdbc14-kube-api-access-8q97z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-88bt7\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.414758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.414595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2341045-6356-4333-b9fb-e3a7adfdbc14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-88bt7\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.414910 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.414895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2341045-6356-4333-b9fb-e3a7adfdbc14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-88bt7\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.429674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.429651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q97z\" (UniqueName: \"kubernetes.io/projected/d2341045-6356-4333-b9fb-e3a7adfdbc14-kube-api-access-8q97z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-88bt7\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.560118 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.560005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:39.888082 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:39.885555    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"]
Sep 15 12:10:39.889373 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:10:39.889339    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2341045_6356_4333_b9fb_e3a7adfdbc14.slice/crio-cf059ea9434729394f871637b9f1531a87929015d15d1d891577d5c0940f18e1 WatchSource:0}: Error finding container cf059ea9434729394f871637b9f1531a87929015d15d1d891577d5c0940f18e1: Status 404 returned error can't find the container with id cf059ea9434729394f871637b9f1531a87929015d15d1d891577d5c0940f18e1
Sep 15 12:10:40.166792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:40.166686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" event={"ID":"d2341045-6356-4333-b9fb-e3a7adfdbc14","Type":"ContainerStarted","Data":"70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"}
Sep 15 12:10:40.166792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:40.166753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" event={"ID":"d2341045-6356-4333-b9fb-e3a7adfdbc14","Type":"ContainerStarted","Data":"cf059ea9434729394f871637b9f1531a87929015d15d1d891577d5c0940f18e1"}
Sep 15 12:10:40.166792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:40.166781    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:40.214643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:40.214591    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" podStartSLOduration=1.214574385 podStartE2EDuration="1.214574385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:40.214340296 +0000 UTC m=+520.910899884" watchObservedRunningTime="2026-09-15 12:10:40.214574385 +0000 UTC m=+520.911133963"
Sep 15 12:10:43.759715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:43.759679    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:43.975659 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:43.975622    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:44.203735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:44.203701    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:44.406907 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:44.406865    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:47.152253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:47.152223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-cs744"
Sep 15 12:10:47.152600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:47.152300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:47.724503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:47.724468    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:47.880597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:47.880560    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:51.173853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.173813    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:10:51.247849 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.247815    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"]
Sep 15 12:10:51.248200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.248169    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9" podUID="5429555d-7a61-4eae-acc1-0c1a4a58ead5" containerName="manager" containerID="cri-o://ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60" gracePeriod=10
Sep 15 12:10:51.374007 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.373973    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:51.494419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.494392    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:51.551545 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.551511    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:51.624106 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.624076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phbrc\" (UniqueName: \"kubernetes.io/projected/5429555d-7a61-4eae-acc1-0c1a4a58ead5-kube-api-access-phbrc\") pod \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") "
Sep 15 12:10:51.624303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.624136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5429555d-7a61-4eae-acc1-0c1a4a58ead5-extensions-socket-volume\") pod \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\" (UID: \"5429555d-7a61-4eae-acc1-0c1a4a58ead5\") "
Sep 15 12:10:51.624543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.624519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5429555d-7a61-4eae-acc1-0c1a4a58ead5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5429555d-7a61-4eae-acc1-0c1a4a58ead5" (UID: "5429555d-7a61-4eae-acc1-0c1a4a58ead5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:10:51.626324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.626294    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5429555d-7a61-4eae-acc1-0c1a4a58ead5-kube-api-access-phbrc" (OuterVolumeSpecName: "kube-api-access-phbrc") pod "5429555d-7a61-4eae-acc1-0c1a4a58ead5" (UID: "5429555d-7a61-4eae-acc1-0c1a4a58ead5"). InnerVolumeSpecName "kube-api-access-phbrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:10:51.647355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.647324    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:10:51.725821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.725735    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbrc\" (UniqueName: \"kubernetes.io/projected/5429555d-7a61-4eae-acc1-0c1a4a58ead5-kube-api-access-phbrc\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:10:51.725821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:51.725767    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5429555d-7a61-4eae-acc1-0c1a4a58ead5-extensions-socket-volume\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:10:52.230090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.230056    2583 generic.go:356] "Generic (PLEG): container finished" podID="5429555d-7a61-4eae-acc1-0c1a4a58ead5" containerID="ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60" exitCode=0
Sep 15 12:10:52.230487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.230114    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"
Sep 15 12:10:52.230487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.230129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9" event={"ID":"5429555d-7a61-4eae-acc1-0c1a4a58ead5","Type":"ContainerDied","Data":"ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60"}
Sep 15 12:10:52.230487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.230172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9" event={"ID":"5429555d-7a61-4eae-acc1-0c1a4a58ead5","Type":"ContainerDied","Data":"c6ca1670a3fdef46571924d348a3708f6ac47e0808658b5944f808f9f7466424"}
Sep 15 12:10:52.230487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.230194    2583 scope.go:117] "RemoveContainer" containerID="ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60"
Sep 15 12:10:52.238593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.238575    2583 scope.go:117] "RemoveContainer" containerID="ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60"
Sep 15 12:10:52.238895 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:10:52.238875    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60\": container with ID starting with ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60 not found: ID does not exist" containerID="ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60"
Sep 15 12:10:52.238961 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.238907    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60"} err="failed to get container status \"ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60\": rpc error: code = NotFound desc = could not find container \"ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60\": container with ID starting with ea51eaca4ee0c03f4716ec6f7c85d4d5eefaad573f1f4077b56996d4e599df60 not found: ID does not exist"
Sep 15 12:10:52.249379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.249356    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"]
Sep 15 12:10:52.256762 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:52.256742    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4vxz9"]
Sep 15 12:10:53.806544 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:10:53.806512    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5429555d-7a61-4eae-acc1-0c1a4a58ead5" path="/var/lib/kubelet/pods/5429555d-7a61-4eae-acc1-0c1a4a58ead5/volumes"
Sep 15 12:11:07.755091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:07.755052    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:08.098276 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:08.098234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:10.141497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.141457    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:10.141919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.141847    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5429555d-7a61-4eae-acc1-0c1a4a58ead5" containerName="manager"
Sep 15 12:11:10.141919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.141860    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5429555d-7a61-4eae-acc1-0c1a4a58ead5" containerName="manager"
Sep 15 12:11:10.141919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.141919    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5429555d-7a61-4eae-acc1-0c1a4a58ead5" containerName="manager"
Sep 15 12:11:10.144793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.144775    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.147107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.147077    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 12:11:10.147107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.147098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n2ln9\""
Sep 15 12:11:10.149973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.149950    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:10.154130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.154108    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:10.230369 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.230338    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:10.282652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.282618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzjh\" (UniqueName: \"kubernetes.io/projected/c2b9ee31-4eee-4412-82e3-011918782cf8-kube-api-access-nzzjh\") pod \"limitador-limitador-8588749977-lxpdz\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") " pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.282800 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.282665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2b9ee31-4eee-4412-82e3-011918782cf8-config-file\") pod \"limitador-limitador-8588749977-lxpdz\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") " pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.383603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.383562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzjh\" (UniqueName: \"kubernetes.io/projected/c2b9ee31-4eee-4412-82e3-011918782cf8-kube-api-access-nzzjh\") pod \"limitador-limitador-8588749977-lxpdz\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") " pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.383603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.383608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2b9ee31-4eee-4412-82e3-011918782cf8-config-file\") pod \"limitador-limitador-8588749977-lxpdz\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") " pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.384230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.384210    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2b9ee31-4eee-4412-82e3-011918782cf8-config-file\") pod \"limitador-limitador-8588749977-lxpdz\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") " pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.391700 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.391643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzjh\" (UniqueName: \"kubernetes.io/projected/c2b9ee31-4eee-4412-82e3-011918782cf8-kube-api-access-nzzjh\") pod \"limitador-limitador-8588749977-lxpdz\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") " pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.455742 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.455702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:10.614119 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.614075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:10.675573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.675492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:10.726323 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:10.726299    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:10.728526 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:10.728498    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b9ee31_4eee_4412_82e3_011918782cf8.slice/crio-59ce43fad2523577e4c3e131739ba07560fe91170d81092fb60befa15dcb5b83 WatchSource:0}: Error finding container 59ce43fad2523577e4c3e131739ba07560fe91170d81092fb60befa15dcb5b83: Status 404 returned error can't find the container with id 59ce43fad2523577e4c3e131739ba07560fe91170d81092fb60befa15dcb5b83
Sep 15 12:11:11.003924 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.003841    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:11.105251 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.105215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:11.322044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.321983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz" event={"ID":"c2b9ee31-4eee-4412-82e3-011918782cf8","Type":"ContainerStarted","Data":"59ce43fad2523577e4c3e131739ba07560fe91170d81092fb60befa15dcb5b83"}
Sep 15 12:11:11.374798 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.374763    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-sfvrg"]
Sep 15 12:11:11.377506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.377486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:11.379856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.379831    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k4g6t\""
Sep 15 12:11:11.397119 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.397090    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sfvrg"]
Sep 15 12:11:11.493465 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.493430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhv5\" (UniqueName: \"kubernetes.io/projected/d6d36c65-fcef-4762-bd59-96dc23fcdbc2-kube-api-access-xzhv5\") pod \"authorino-59c974bd48-sfvrg\" (UID: \"d6d36c65-fcef-4762-bd59-96dc23fcdbc2\") " pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:11.594501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.594407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhv5\" (UniqueName: \"kubernetes.io/projected/d6d36c65-fcef-4762-bd59-96dc23fcdbc2-kube-api-access-xzhv5\") pod \"authorino-59c974bd48-sfvrg\" (UID: \"d6d36c65-fcef-4762-bd59-96dc23fcdbc2\") " pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:11.607312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.607268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhv5\" (UniqueName: \"kubernetes.io/projected/d6d36c65-fcef-4762-bd59-96dc23fcdbc2-kube-api-access-xzhv5\") pod \"authorino-59c974bd48-sfvrg\" (UID: \"d6d36c65-fcef-4762-bd59-96dc23fcdbc2\") " pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:11.687767 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.687731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:11.957544 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:11.957514    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sfvrg"]
Sep 15 12:11:11.960076 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:11.960040    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6d36c65_fcef_4762_bd59_96dc23fcdbc2.slice/crio-3d5b731a190e8e4a07c5cb3af1627503dd9c368d22805d809796adf1253a62de WatchSource:0}: Error finding container 3d5b731a190e8e4a07c5cb3af1627503dd9c368d22805d809796adf1253a62de: Status 404 returned error can't find the container with id 3d5b731a190e8e4a07c5cb3af1627503dd9c368d22805d809796adf1253a62de
Sep 15 12:11:12.328547 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:12.328516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz" event={"ID":"c2b9ee31-4eee-4412-82e3-011918782cf8","Type":"ContainerStarted","Data":"df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d"}
Sep 15 12:11:12.328922 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:12.328621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:12.329676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:12.329655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sfvrg" event={"ID":"d6d36c65-fcef-4762-bd59-96dc23fcdbc2","Type":"ContainerStarted","Data":"3d5b731a190e8e4a07c5cb3af1627503dd9c368d22805d809796adf1253a62de"}
Sep 15 12:11:12.347588 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:12.346837    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz" podStartSLOduration=0.823141028 podStartE2EDuration="2.346820502s" totalImagesPullingTime="1.523679474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:12.346757894 +0000 UTC m=+553.043317491" watchObservedRunningTime="2026-09-15 12:11:12.346820502 +0000 UTC m=+553.043380079"
Sep 15 12:11:12.648525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:12.648491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:12.754913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:12.754873    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:14.342351 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:14.342302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sfvrg" event={"ID":"d6d36c65-fcef-4762-bd59-96dc23fcdbc2","Type":"ContainerStarted","Data":"5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a"}
Sep 15 12:11:14.358259 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:14.358206    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-sfvrg" podStartSLOduration=1.6250436700000002 podStartE2EDuration="3.358189998s" totalImagesPullingTime="1.733146328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:14.3572209 +0000 UTC m=+555.053780510" watchObservedRunningTime="2026-09-15 12:11:14.358189998 +0000 UTC m=+555.054749576"
Sep 15 12:11:23.336676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:23.336642    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:25.280619 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.280581    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:25.281059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.280768    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz" podUID="c2b9ee31-4eee-4412-82e3-011918782cf8" containerName="limitador" containerID="cri-o://df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d" gracePeriod=30
Sep 15 12:11:25.824890 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.824861    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:25.919796 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.919704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzzjh\" (UniqueName: \"kubernetes.io/projected/c2b9ee31-4eee-4412-82e3-011918782cf8-kube-api-access-nzzjh\") pod \"c2b9ee31-4eee-4412-82e3-011918782cf8\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") "
Sep 15 12:11:25.919796 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.919744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2b9ee31-4eee-4412-82e3-011918782cf8-config-file\") pod \"c2b9ee31-4eee-4412-82e3-011918782cf8\" (UID: \"c2b9ee31-4eee-4412-82e3-011918782cf8\") "
Sep 15 12:11:25.920215 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.920188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b9ee31-4eee-4412-82e3-011918782cf8-config-file" (OuterVolumeSpecName: "config-file") pod "c2b9ee31-4eee-4412-82e3-011918782cf8" (UID: "c2b9ee31-4eee-4412-82e3-011918782cf8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:11:25.921857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:25.921834    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b9ee31-4eee-4412-82e3-011918782cf8-kube-api-access-nzzjh" (OuterVolumeSpecName: "kube-api-access-nzzjh") pod "c2b9ee31-4eee-4412-82e3-011918782cf8" (UID: "c2b9ee31-4eee-4412-82e3-011918782cf8"). InnerVolumeSpecName "kube-api-access-nzzjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:26.020701 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.020666    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzjh\" (UniqueName: \"kubernetes.io/projected/c2b9ee31-4eee-4412-82e3-011918782cf8-kube-api-access-nzzjh\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:11:26.020701 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.020696    2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2b9ee31-4eee-4412-82e3-011918782cf8-config-file\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:11:26.404047 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.403995    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2b9ee31-4eee-4412-82e3-011918782cf8" containerID="df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d" exitCode=0
Sep 15 12:11:26.404569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.404063    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz"
Sep 15 12:11:26.404569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.404089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz" event={"ID":"c2b9ee31-4eee-4412-82e3-011918782cf8","Type":"ContainerDied","Data":"df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d"}
Sep 15 12:11:26.404569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.404152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lxpdz" event={"ID":"c2b9ee31-4eee-4412-82e3-011918782cf8","Type":"ContainerDied","Data":"59ce43fad2523577e4c3e131739ba07560fe91170d81092fb60befa15dcb5b83"}
Sep 15 12:11:26.404569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.404184    2583 scope.go:117] "RemoveContainer" containerID="df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d"
Sep 15 12:11:26.413302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.413287    2583 scope.go:117] "RemoveContainer" containerID="df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d"
Sep 15 12:11:26.413547 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:26.413532    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d\": container with ID starting with df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d not found: ID does not exist" containerID="df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d"
Sep 15 12:11:26.413590 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.413554    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d"} err="failed to get container status \"df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d\": rpc error: code = NotFound desc = could not find container \"df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d\": container with ID starting with df681278354436f7925bd5135fca8ddeeec21cc2843df37f32a3916444dd032d not found: ID does not exist"
Sep 15 12:11:26.426405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.426380    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:26.430431 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:26.430410    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lxpdz"]
Sep 15 12:11:27.806033 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:27.805986    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b9ee31-4eee-4412-82e3-011918782cf8" path="/var/lib/kubelet/pods/c2b9ee31-4eee-4412-82e3-011918782cf8/volumes"
Sep 15 12:11:31.388680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.388624    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:31.759152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.759009    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:31.828183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.828155    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"]
Sep 15 12:11:31.828509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.828497    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b9ee31-4eee-4412-82e3-011918782cf8" containerName="limitador"
Sep 15 12:11:31.828555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.828511    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b9ee31-4eee-4412-82e3-011918782cf8" containerName="limitador"
Sep 15 12:11:31.828588 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.828581    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b9ee31-4eee-4412-82e3-011918782cf8" containerName="limitador"
Sep 15 12:11:31.833105 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.833086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:31.835439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.835405    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:11:31.835565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.835518    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 15 12:11:31.836294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.836277    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6wd9r\""
Sep 15 12:11:31.836410 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.836280    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 12:11:31.840851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.840825    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"]
Sep 15 12:11:31.866195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.866166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf73b-ec28-4397-8eb5-a71b267064e9-data\") pod \"postgres-69fb4594cf-lmpjp\" (UID: \"f93cf73b-ec28-4397-8eb5-a71b267064e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:31.866346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.866214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g7h4\" (UniqueName: \"kubernetes.io/projected/f93cf73b-ec28-4397-8eb5-a71b267064e9-kube-api-access-6g7h4\") pod \"postgres-69fb4594cf-lmpjp\" (UID: \"f93cf73b-ec28-4397-8eb5-a71b267064e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:31.875671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.875626    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:31.967118 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.967078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g7h4\" (UniqueName: \"kubernetes.io/projected/f93cf73b-ec28-4397-8eb5-a71b267064e9-kube-api-access-6g7h4\") pod \"postgres-69fb4594cf-lmpjp\" (UID: \"f93cf73b-ec28-4397-8eb5-a71b267064e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:31.967269 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.967199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf73b-ec28-4397-8eb5-a71b267064e9-data\") pod \"postgres-69fb4594cf-lmpjp\" (UID: \"f93cf73b-ec28-4397-8eb5-a71b267064e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:31.967602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.967584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf73b-ec28-4397-8eb5-a71b267064e9-data\") pod \"postgres-69fb4594cf-lmpjp\" (UID: \"f93cf73b-ec28-4397-8eb5-a71b267064e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:31.975294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:31.975273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g7h4\" (UniqueName: \"kubernetes.io/projected/f93cf73b-ec28-4397-8eb5-a71b267064e9-kube-api-access-6g7h4\") pod \"postgres-69fb4594cf-lmpjp\" (UID: \"f93cf73b-ec28-4397-8eb5-a71b267064e9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:32.143953 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:32.143916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:32.417146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:32.417115    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"]
Sep 15 12:11:32.419286 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:32.419260    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93cf73b_ec28_4397_8eb5_a71b267064e9.slice/crio-b8394a54cc090d8e97c47256dbf95c4dd88637e49efd44b38e55ee1cdc47e3b3 WatchSource:0}: Error finding container b8394a54cc090d8e97c47256dbf95c4dd88637e49efd44b38e55ee1cdc47e3b3: Status 404 returned error can't find the container with id b8394a54cc090d8e97c47256dbf95c4dd88637e49efd44b38e55ee1cdc47e3b3
Sep 15 12:11:32.433272 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:32.433237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp" event={"ID":"f93cf73b-ec28-4397-8eb5-a71b267064e9","Type":"ContainerStarted","Data":"b8394a54cc090d8e97c47256dbf95c4dd88637e49efd44b38e55ee1cdc47e3b3"}
Sep 15 12:11:39.474718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:39.474680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp" event={"ID":"f93cf73b-ec28-4397-8eb5-a71b267064e9","Type":"ContainerStarted","Data":"1782a6778005c5c83f72c6d51e8fd913b48f6e71a2ac8416c25d15e5d70b9d19"}
Sep 15 12:11:39.475135 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:39.474757    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:39.492807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:39.492755    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp" podStartSLOduration=1.641577815 podStartE2EDuration="8.492740518s" totalImagesPullingTime="6.851162703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:39.490403493 +0000 UTC m=+580.186963070" watchObservedRunningTime="2026-09-15 12:11:39.492740518 +0000 UTC m=+580.189300096"
Sep 15 12:11:45.509177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:45.509137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lmpjp"
Sep 15 12:11:45.830345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:45.830307    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:46.061583 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.061547    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-wjscr"]
Sep 15 12:11:46.067422 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.067396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:46.071794 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.071761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-wjscr"]
Sep 15 12:11:46.192992 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.192885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznld\" (UniqueName: \"kubernetes.io/projected/510116da-f3d9-45c1-a4e3-5214f18b2ad9-kube-api-access-wznld\") pod \"authorino-557f876f69-wjscr\" (UID: \"510116da-f3d9-45c1-a4e3-5214f18b2ad9\") " pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:46.293848 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.293812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznld\" (UniqueName: \"kubernetes.io/projected/510116da-f3d9-45c1-a4e3-5214f18b2ad9-kube-api-access-wznld\") pod \"authorino-557f876f69-wjscr\" (UID: \"510116da-f3d9-45c1-a4e3-5214f18b2ad9\") " pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:46.313034 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.312989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznld\" (UniqueName: \"kubernetes.io/projected/510116da-f3d9-45c1-a4e3-5214f18b2ad9-kube-api-access-wznld\") pod \"authorino-557f876f69-wjscr\" (UID: \"510116da-f3d9-45c1-a4e3-5214f18b2ad9\") " pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:46.338473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.338434    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wjscr"]
Sep 15 12:11:46.338729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.338712    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:46.389500 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.389449    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f7cf647cd-w87xf"]
Sep 15 12:11:46.395971 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.395940    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:46.400255 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.400177    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7cf647cd-w87xf"]
Sep 15 12:11:46.495651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.495562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmk8w\" (UniqueName: \"kubernetes.io/projected/60315825-2c53-4ee1-b1f7-1c3cc11dd819-kube-api-access-cmk8w\") pod \"authorino-5f7cf647cd-w87xf\" (UID: \"60315825-2c53-4ee1-b1f7-1c3cc11dd819\") " pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:46.596445 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.596413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmk8w\" (UniqueName: \"kubernetes.io/projected/60315825-2c53-4ee1-b1f7-1c3cc11dd819-kube-api-access-cmk8w\") pod \"authorino-5f7cf647cd-w87xf\" (UID: \"60315825-2c53-4ee1-b1f7-1c3cc11dd819\") " pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:46.605274 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.605247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmk8w\" (UniqueName: \"kubernetes.io/projected/60315825-2c53-4ee1-b1f7-1c3cc11dd819-kube-api-access-cmk8w\") pod \"authorino-5f7cf647cd-w87xf\" (UID: \"60315825-2c53-4ee1-b1f7-1c3cc11dd819\") " pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:46.614752 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.614724    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wjscr"]
Sep 15 12:11:46.617200 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:46.617155    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod510116da_f3d9_45c1_a4e3_5214f18b2ad9.slice/crio-9f1bbf79e56bf90b90e8184da5f4c6213b3bd557a87ed6518e0001d96a0b639b WatchSource:0}: Error finding container 9f1bbf79e56bf90b90e8184da5f4c6213b3bd557a87ed6518e0001d96a0b639b: Status 404 returned error can't find the container with id 9f1bbf79e56bf90b90e8184da5f4c6213b3bd557a87ed6518e0001d96a0b639b
Sep 15 12:11:46.662540 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.662496    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:46.715672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.715636    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:46.988259 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:46.988231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7cf647cd-w87xf"]
Sep 15 12:11:46.990075 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:46.990048    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60315825_2c53_4ee1_b1f7_1c3cc11dd819.slice/crio-8824907198eb597b254d47f280da422bf19bdded6e41ba8b31f1007f09d5266f WatchSource:0}: Error finding container 8824907198eb597b254d47f280da422bf19bdded6e41ba8b31f1007f09d5266f: Status 404 returned error can't find the container with id 8824907198eb597b254d47f280da422bf19bdded6e41ba8b31f1007f09d5266f
Sep 15 12:11:47.515917 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.515814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7cf647cd-w87xf" event={"ID":"60315825-2c53-4ee1-b1f7-1c3cc11dd819","Type":"ContainerStarted","Data":"c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4"}
Sep 15 12:11:47.515917 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.515861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7cf647cd-w87xf" event={"ID":"60315825-2c53-4ee1-b1f7-1c3cc11dd819","Type":"ContainerStarted","Data":"8824907198eb597b254d47f280da422bf19bdded6e41ba8b31f1007f09d5266f"}
Sep 15 12:11:47.517520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.517491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-wjscr" event={"ID":"510116da-f3d9-45c1-a4e3-5214f18b2ad9","Type":"ContainerStarted","Data":"bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a"}
Sep 15 12:11:47.517520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.517525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-wjscr" event={"ID":"510116da-f3d9-45c1-a4e3-5214f18b2ad9","Type":"ContainerStarted","Data":"9f1bbf79e56bf90b90e8184da5f4c6213b3bd557a87ed6518e0001d96a0b639b"}
Sep 15 12:11:47.517666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.517560    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-wjscr" podUID="510116da-f3d9-45c1-a4e3-5214f18b2ad9" containerName="authorino" containerID="cri-o://bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a" gracePeriod=30
Sep 15 12:11:47.534722 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.534672    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f7cf647cd-w87xf" podStartSLOduration=1.190583436 podStartE2EDuration="1.53465703s" totalImagesPullingTime="344.073594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:47.533090439 +0000 UTC m=+588.229650017" watchObservedRunningTime="2026-09-15 12:11:47.53465703 +0000 UTC m=+588.231216607"
Sep 15 12:11:47.552554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.552485    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-wjscr" podStartSLOduration=1.024380389 podStartE2EDuration="1.55246115s" totalImagesPullingTime="528.080761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:47.550483205 +0000 UTC m=+588.247042783" watchObservedRunningTime="2026-09-15 12:11:47.55246115 +0000 UTC m=+588.249020729"
Sep 15 12:11:47.568722 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.568679    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sfvrg"]
Sep 15 12:11:47.569135 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.568871    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-sfvrg" podUID="d6d36c65-fcef-4762-bd59-96dc23fcdbc2" containerName="authorino" containerID="cri-o://5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a" gracePeriod=30
Sep 15 12:11:47.661580 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.661541    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cbf5dfc84-9rs89"]
Sep 15 12:11:47.666051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.666002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.669762 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.669738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 12:11:47.675750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.675719    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cbf5dfc84-9rs89"]
Sep 15 12:11:47.706198 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.706160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krq8c\" (UniqueName: \"kubernetes.io/projected/b404102f-059c-4575-b8b1-7fcd81b95d62-kube-api-access-krq8c\") pod \"authorino-5cbf5dfc84-9rs89\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") " pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.706364 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.706303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b404102f-059c-4575-b8b1-7fcd81b95d62-tls-cert\") pod \"authorino-5cbf5dfc84-9rs89\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") " pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.807495 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.807456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b404102f-059c-4575-b8b1-7fcd81b95d62-tls-cert\") pod \"authorino-5cbf5dfc84-9rs89\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") " pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.807686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.807558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krq8c\" (UniqueName: \"kubernetes.io/projected/b404102f-059c-4575-b8b1-7fcd81b95d62-kube-api-access-krq8c\") pod \"authorino-5cbf5dfc84-9rs89\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") " pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.810073 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.810047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b404102f-059c-4575-b8b1-7fcd81b95d62-tls-cert\") pod \"authorino-5cbf5dfc84-9rs89\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") " pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.815378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.815354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krq8c\" (UniqueName: \"kubernetes.io/projected/b404102f-059c-4575-b8b1-7fcd81b95d62-kube-api-access-krq8c\") pod \"authorino-5cbf5dfc84-9rs89\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") " pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:47.829868 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.829828    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:47.837569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.837543    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:47.908825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.908783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznld\" (UniqueName: \"kubernetes.io/projected/510116da-f3d9-45c1-a4e3-5214f18b2ad9-kube-api-access-wznld\") pod \"510116da-f3d9-45c1-a4e3-5214f18b2ad9\" (UID: \"510116da-f3d9-45c1-a4e3-5214f18b2ad9\") "
Sep 15 12:11:47.908825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.908830    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzhv5\" (UniqueName: \"kubernetes.io/projected/d6d36c65-fcef-4762-bd59-96dc23fcdbc2-kube-api-access-xzhv5\") pod \"d6d36c65-fcef-4762-bd59-96dc23fcdbc2\" (UID: \"d6d36c65-fcef-4762-bd59-96dc23fcdbc2\") "
Sep 15 12:11:47.916143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.911832    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d36c65-fcef-4762-bd59-96dc23fcdbc2-kube-api-access-xzhv5" (OuterVolumeSpecName: "kube-api-access-xzhv5") pod "d6d36c65-fcef-4762-bd59-96dc23fcdbc2" (UID: "d6d36c65-fcef-4762-bd59-96dc23fcdbc2"). InnerVolumeSpecName "kube-api-access-xzhv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:47.916143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.911912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510116da-f3d9-45c1-a4e3-5214f18b2ad9-kube-api-access-wznld" (OuterVolumeSpecName: "kube-api-access-wznld") pod "510116da-f3d9-45c1-a4e3-5214f18b2ad9" (UID: "510116da-f3d9-45c1-a4e3-5214f18b2ad9"). InnerVolumeSpecName "kube-api-access-wznld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:47.977416 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:47.977377    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:11:48.010758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.010715    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznld\" (UniqueName: \"kubernetes.io/projected/510116da-f3d9-45c1-a4e3-5214f18b2ad9-kube-api-access-wznld\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:11:48.010758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.010761    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhv5\" (UniqueName: \"kubernetes.io/projected/d6d36c65-fcef-4762-bd59-96dc23fcdbc2-kube-api-access-xzhv5\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:11:48.253432 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.253405    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cbf5dfc84-9rs89"]
Sep 15 12:11:48.255506 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:48.255476    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb404102f_059c_4575_b8b1_7fcd81b95d62.slice/crio-d4c9469cbeee1deb2fd8a135db0c69eae03932f4c9117970645890ce941170d9 WatchSource:0}: Error finding container d4c9469cbeee1deb2fd8a135db0c69eae03932f4c9117970645890ce941170d9: Status 404 returned error can't find the container with id d4c9469cbeee1deb2fd8a135db0c69eae03932f4c9117970645890ce941170d9
Sep 15 12:11:48.524338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.524303    2583 generic.go:356] "Generic (PLEG): container finished" podID="510116da-f3d9-45c1-a4e3-5214f18b2ad9" containerID="bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a" exitCode=0
Sep 15 12:11:48.524542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.524357    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wjscr"
Sep 15 12:11:48.524542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.524425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-wjscr" event={"ID":"510116da-f3d9-45c1-a4e3-5214f18b2ad9","Type":"ContainerDied","Data":"bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a"}
Sep 15 12:11:48.524542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.524499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-wjscr" event={"ID":"510116da-f3d9-45c1-a4e3-5214f18b2ad9","Type":"ContainerDied","Data":"9f1bbf79e56bf90b90e8184da5f4c6213b3bd557a87ed6518e0001d96a0b639b"}
Sep 15 12:11:48.524542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.524529    2583 scope.go:117] "RemoveContainer" containerID="bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a"
Sep 15 12:11:48.525893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.525786    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6d36c65-fcef-4762-bd59-96dc23fcdbc2" containerID="5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a" exitCode=0
Sep 15 12:11:48.525893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.525856    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sfvrg" event={"ID":"d6d36c65-fcef-4762-bd59-96dc23fcdbc2","Type":"ContainerDied","Data":"5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a"}
Sep 15 12:11:48.525893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.525886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sfvrg" event={"ID":"d6d36c65-fcef-4762-bd59-96dc23fcdbc2","Type":"ContainerDied","Data":"3d5b731a190e8e4a07c5cb3af1627503dd9c368d22805d809796adf1253a62de"}
Sep 15 12:11:48.526075 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.525860    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sfvrg"
Sep 15 12:11:48.527371 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.527331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89" event={"ID":"b404102f-059c-4575-b8b1-7fcd81b95d62","Type":"ContainerStarted","Data":"d4c9469cbeee1deb2fd8a135db0c69eae03932f4c9117970645890ce941170d9"}
Sep 15 12:11:48.533859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.533779    2583 scope.go:117] "RemoveContainer" containerID="bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a"
Sep 15 12:11:48.534138 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:48.534116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a\": container with ID starting with bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a not found: ID does not exist" containerID="bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a"
Sep 15 12:11:48.534225 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.534146    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a"} err="failed to get container status \"bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a\": rpc error: code = NotFound desc = could not find container \"bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a\": container with ID starting with bbaf4c5c7b18241893b0ee4ca128ded669d53060da2c92fafee7205867cbf03a not found: ID does not exist"
Sep 15 12:11:48.534225 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.534162    2583 scope.go:117] "RemoveContainer" containerID="5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a"
Sep 15 12:11:48.542220 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.542199    2583 scope.go:117] "RemoveContainer" containerID="5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a"
Sep 15 12:11:48.542507 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:48.542487    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a\": container with ID starting with 5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a not found: ID does not exist" containerID="5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a"
Sep 15 12:11:48.542568 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.542519    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a"} err="failed to get container status \"5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a\": rpc error: code = NotFound desc = could not find container \"5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a\": container with ID starting with 5f0b4ba38710ca54607d8a118371a3448c5aa1d3b99036c1178f832e071b213a not found: ID does not exist"
Sep 15 12:11:48.549434 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.549406    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wjscr"]
Sep 15 12:11:48.553878 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.553855    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-wjscr"]
Sep 15 12:11:48.564107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.564072    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sfvrg"]
Sep 15 12:11:48.566573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:48.566548    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sfvrg"]
Sep 15 12:11:49.534389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.534339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89" event={"ID":"b404102f-059c-4575-b8b1-7fcd81b95d62","Type":"ContainerStarted","Data":"097be6b89d8275f89618f92c08e51ca5ca6a5cfd1632db9ea2e0650b2461ff84"}
Sep 15 12:11:49.551217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.551165    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89" podStartSLOduration=2.221795318 podStartE2EDuration="2.5511484s" totalImagesPullingTime="329.353082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:49.549858921 +0000 UTC m=+590.246418498" watchObservedRunningTime="2026-09-15 12:11:49.5511484 +0000 UTC m=+590.247707978"
Sep 15 12:11:49.579119 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.579081    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7cf647cd-w87xf"]
Sep 15 12:11:49.579311 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.579237    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f7cf647cd-w87xf" podUID="60315825-2c53-4ee1-b1f7-1c3cc11dd819" containerName="authorino" containerID="cri-o://c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4" gracePeriod=30
Sep 15 12:11:49.807679 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.807598    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510116da-f3d9-45c1-a4e3-5214f18b2ad9" path="/var/lib/kubelet/pods/510116da-f3d9-45c1-a4e3-5214f18b2ad9/volumes"
Sep 15 12:11:49.808094 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.808076    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6d36c65-fcef-4762-bd59-96dc23fcdbc2" path="/var/lib/kubelet/pods/d6d36c65-fcef-4762-bd59-96dc23fcdbc2/volumes"
Sep 15 12:11:49.833351 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.833322    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:49.929012 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.928984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmk8w\" (UniqueName: \"kubernetes.io/projected/60315825-2c53-4ee1-b1f7-1c3cc11dd819-kube-api-access-cmk8w\") pod \"60315825-2c53-4ee1-b1f7-1c3cc11dd819\" (UID: \"60315825-2c53-4ee1-b1f7-1c3cc11dd819\") "
Sep 15 12:11:49.931175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:49.931144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60315825-2c53-4ee1-b1f7-1c3cc11dd819-kube-api-access-cmk8w" (OuterVolumeSpecName: "kube-api-access-cmk8w") pod "60315825-2c53-4ee1-b1f7-1c3cc11dd819" (UID: "60315825-2c53-4ee1-b1f7-1c3cc11dd819"). InnerVolumeSpecName "kube-api-access-cmk8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:11:50.030493 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.030457    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmk8w\" (UniqueName: \"kubernetes.io/projected/60315825-2c53-4ee1-b1f7-1c3cc11dd819-kube-api-access-cmk8w\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:11:50.541305 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.541268    2583 generic.go:356] "Generic (PLEG): container finished" podID="60315825-2c53-4ee1-b1f7-1c3cc11dd819" containerID="c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4" exitCode=0
Sep 15 12:11:50.541801 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.541313    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7cf647cd-w87xf"
Sep 15 12:11:50.541801 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.541349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7cf647cd-w87xf" event={"ID":"60315825-2c53-4ee1-b1f7-1c3cc11dd819","Type":"ContainerDied","Data":"c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4"}
Sep 15 12:11:50.541801 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.541397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7cf647cd-w87xf" event={"ID":"60315825-2c53-4ee1-b1f7-1c3cc11dd819","Type":"ContainerDied","Data":"8824907198eb597b254d47f280da422bf19bdded6e41ba8b31f1007f09d5266f"}
Sep 15 12:11:50.541801 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.541423    2583 scope.go:117] "RemoveContainer" containerID="c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4"
Sep 15 12:11:50.550088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.550067    2583 scope.go:117] "RemoveContainer" containerID="c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4"
Sep 15 12:11:50.550409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:50.550393    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4\": container with ID starting with c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4 not found: ID does not exist" containerID="c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4"
Sep 15 12:11:50.550466 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.550416    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4"} err="failed to get container status \"c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4\": rpc error: code = NotFound desc = could not find container \"c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4\": container with ID starting with c573b1e1547cf65a27a2f91ffb9cd9264602ef2fab3bb8405ea7138b1be564d4 not found: ID does not exist"
Sep 15 12:11:50.566389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.566355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7cf647cd-w87xf"]
Sep 15 12:11:50.572248 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:50.572221    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f7cf647cd-w87xf"]
Sep 15 12:11:51.806851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:51.806815    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60315825-2c53-4ee1-b1f7-1c3cc11dd819" path="/var/lib/kubelet/pods/60315825-2c53-4ee1-b1f7-1c3cc11dd819/volumes"
Sep 15 12:11:52.370923 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.370887    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:52.409743 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.409701    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76dbd9ddcd-t4wv4"]
Sep 15 12:11:52.410218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410187    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6d36c65-fcef-4762-bd59-96dc23fcdbc2" containerName="authorino"
Sep 15 12:11:52.410218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410219    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6d36c65-fcef-4762-bd59-96dc23fcdbc2" containerName="authorino"
Sep 15 12:11:52.410389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410235    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60315825-2c53-4ee1-b1f7-1c3cc11dd819" containerName="authorino"
Sep 15 12:11:52.410389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410243    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="60315825-2c53-4ee1-b1f7-1c3cc11dd819" containerName="authorino"
Sep 15 12:11:52.410389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410281    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="510116da-f3d9-45c1-a4e3-5214f18b2ad9" containerName="authorino"
Sep 15 12:11:52.410389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410290    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="510116da-f3d9-45c1-a4e3-5214f18b2ad9" containerName="authorino"
Sep 15 12:11:52.410389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410381    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="60315825-2c53-4ee1-b1f7-1c3cc11dd819" containerName="authorino"
Sep 15 12:11:52.410617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410398    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="510116da-f3d9-45c1-a4e3-5214f18b2ad9" containerName="authorino"
Sep 15 12:11:52.410617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.410407    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6d36c65-fcef-4762-bd59-96dc23fcdbc2" containerName="authorino"
Sep 15 12:11:52.414992 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.414970    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.417481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.417459    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 12:11:52.417859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.417842    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pb5qc\""
Sep 15 12:11:52.418291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.418273    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 12:11:52.418375 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.418339    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 12:11:52.426573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.426548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76dbd9ddcd-t4wv4"]
Sep 15 12:11:52.466141 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.466084    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:52.553748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.553711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.553968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.553837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.553968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.553902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/5cfc6911-0010-445e-96b1-c908a83e81c1-kube-api-access-xjfp2\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.562400 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.562372    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:52.602610 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.602579    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-4cddr"]
Sep 15 12:11:52.606315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.606291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.616899 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.616868    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-4cddr"]
Sep 15 12:11:52.655378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.655298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.655378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.655344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.655545 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.655403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.655545 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.655430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz92s\" (UniqueName: \"kubernetes.io/projected/fbf42d58-7d87-404c-b210-c4fd9abced30-kube-api-access-jz92s\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.655545 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.655454    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 15 12:11:52.655545 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.655524    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 15 12:11:52.655729 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.655569    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert podName:5cfc6911-0010-445e-96b1-c908a83e81c1 nodeName:}" failed. No retries permitted until 2026-09-15 12:11:53.155548041 +0000 UTC m=+593.852107597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert") pod "maas-controller-76dbd9ddcd-t4wv4" (UID: "5cfc6911-0010-445e-96b1-c908a83e81c1") : secret "maas-controller-webhook-cert" not found
Sep 15 12:11:52.655729 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.655591    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs podName:5cfc6911-0010-445e-96b1-c908a83e81c1 nodeName:}" failed. No retries permitted until 2026-09-15 12:11:53.155582272 +0000 UTC m=+593.852141829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs") pod "maas-controller-76dbd9ddcd-t4wv4" (UID: "5cfc6911-0010-445e-96b1-c908a83e81c1") : secret "maas-controller-metrics-tls" not found
Sep 15 12:11:52.655729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.655636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/5cfc6911-0010-445e-96b1-c908a83e81c1-kube-api-access-xjfp2\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.655729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.655667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.664616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.664584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/5cfc6911-0010-445e-96b1-c908a83e81c1-kube-api-access-xjfp2\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:52.756174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.756137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.756341 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.756281    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 15 12:11:52.756341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.756301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz92s\" (UniqueName: \"kubernetes.io/projected/fbf42d58-7d87-404c-b210-c4fd9abced30-kube-api-access-jz92s\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.756472 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.756351    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs podName:fbf42d58-7d87-404c-b210-c4fd9abced30 nodeName:}" failed. No retries permitted until 2026-09-15 12:11:53.256332762 +0000 UTC m=+593.952892319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs") pod "maas-controller-bc7f7d7c4-4cddr" (UID: "fbf42d58-7d87-404c-b210-c4fd9abced30") : secret "maas-controller-metrics-tls" not found
Sep 15 12:11:52.756472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.756403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.756576 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.756489    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 15 12:11:52.756576 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:11:52.756520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert podName:fbf42d58-7d87-404c-b210-c4fd9abced30 nodeName:}" failed. No retries permitted until 2026-09-15 12:11:53.256511702 +0000 UTC m=+593.953071259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert") pod "maas-controller-bc7f7d7c4-4cddr" (UID: "fbf42d58-7d87-404c-b210-c4fd9abced30") : secret "maas-controller-webhook-cert" not found
Sep 15 12:11:52.767649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.767615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz92s\" (UniqueName: \"kubernetes.io/projected/fbf42d58-7d87-404c-b210-c4fd9abced30-kube-api-access-jz92s\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:52.892125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:52.892084    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:53.018583 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.018486    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:53.160681 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.160646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:53.160853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.160713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:53.163169 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.163137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:53.163282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.163202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs\") pod \"maas-controller-76dbd9ddcd-t4wv4\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") " pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:53.261683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.261630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:53.261683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.261699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:53.264284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.264256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:53.264390 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.264303    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert\") pod \"maas-controller-bc7f7d7c4-4cddr\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") " pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:53.326926 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.326889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:53.518159 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.518122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:53.598317 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.598293    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76dbd9ddcd-t4wv4"]
Sep 15 12:11:53.600922 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:53.600894    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfc6911_0010_445e_96b1_c908a83e81c1.slice/crio-fc31ba10fadb60eb0278826cb9830d6c999ec83f47e2e5c6b03fd81962dea45e WatchSource:0}: Error finding container fc31ba10fadb60eb0278826cb9830d6c999ec83f47e2e5c6b03fd81962dea45e: Status 404 returned error can't find the container with id fc31ba10fadb60eb0278826cb9830d6c999ec83f47e2e5c6b03fd81962dea45e
Sep 15 12:11:53.794082 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.794049    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-4cddr"]
Sep 15 12:11:53.797581 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:11:53.797552    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf42d58_7d87_404c_b210_c4fd9abced30.slice/crio-c09f92f43bf349c3462e7b8bbff911fed5e1a4981798a2600f817d7b143c0432 WatchSource:0}: Error finding container c09f92f43bf349c3462e7b8bbff911fed5e1a4981798a2600f817d7b143c0432: Status 404 returned error can't find the container with id c09f92f43bf349c3462e7b8bbff911fed5e1a4981798a2600f817d7b143c0432
Sep 15 12:11:53.939409 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:53.939310    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:54.003543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:54.003507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:54.575726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:54.575685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4" event={"ID":"5cfc6911-0010-445e-96b1-c908a83e81c1","Type":"ContainerStarted","Data":"fc31ba10fadb60eb0278826cb9830d6c999ec83f47e2e5c6b03fd81962dea45e"}
Sep 15 12:11:54.578551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:54.578514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr" event={"ID":"fbf42d58-7d87-404c-b210-c4fd9abced30","Type":"ContainerStarted","Data":"c09f92f43bf349c3462e7b8bbff911fed5e1a4981798a2600f817d7b143c0432"}
Sep 15 12:11:56.504320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.504164    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:56.590740 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.590696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr" event={"ID":"fbf42d58-7d87-404c-b210-c4fd9abced30","Type":"ContainerStarted","Data":"2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15"}
Sep 15 12:11:56.590942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.590912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:11:56.592250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.592216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4" event={"ID":"5cfc6911-0010-445e-96b1-c908a83e81c1","Type":"ContainerStarted","Data":"ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc"}
Sep 15 12:11:56.592357 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.592281    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:11:56.599731 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.599706    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:11:56.615912 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.615868    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr" podStartSLOduration=2.138496999 podStartE2EDuration="4.615855617s" totalImagesPullingTime="2.477358618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:56.614142453 +0000 UTC m=+597.310702051" watchObservedRunningTime="2026-09-15 12:11:56.615855617 +0000 UTC m=+597.312415194"
Sep 15 12:11:56.639303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:56.639248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4" podStartSLOduration=1.975582729 podStartE2EDuration="4.639231649s" totalImagesPullingTime="2.66364892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:11:56.63721369 +0000 UTC m=+597.333773267" watchObservedRunningTime="2026-09-15 12:11:56.639231649 +0000 UTC m=+597.335791227"
Sep 15 12:11:59.736968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:59.736942    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:11:59.739598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:11:59.739574    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:12:07.600409 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.600358    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:12:07.602886 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.600591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:12:07.664543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.664504    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76dbd9ddcd-t4wv4"]
Sep 15 12:12:07.664732 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.664707    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4" podUID="5cfc6911-0010-445e-96b1-c908a83e81c1" containerName="manager" containerID="cri-o://ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc" gracePeriod=30
Sep 15 12:12:07.914936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.914904    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:12:07.991733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.991697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs\") pod \"5cfc6911-0010-445e-96b1-c908a83e81c1\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") "
Sep 15 12:12:07.991897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.991746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert\") pod \"5cfc6911-0010-445e-96b1-c908a83e81c1\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") "
Sep 15 12:12:07.991897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.991787    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/5cfc6911-0010-445e-96b1-c908a83e81c1-kube-api-access-xjfp2\") pod \"5cfc6911-0010-445e-96b1-c908a83e81c1\" (UID: \"5cfc6911-0010-445e-96b1-c908a83e81c1\") "
Sep 15 12:12:07.994054 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.994008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5cfc6911-0010-445e-96b1-c908a83e81c1" (UID: "5cfc6911-0010-445e-96b1-c908a83e81c1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:12:07.994054 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.994042    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5cfc6911-0010-445e-96b1-c908a83e81c1" (UID: "5cfc6911-0010-445e-96b1-c908a83e81c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:12:07.994197 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:07.994085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfc6911-0010-445e-96b1-c908a83e81c1-kube-api-access-xjfp2" (OuterVolumeSpecName: "kube-api-access-xjfp2") pod "5cfc6911-0010-445e-96b1-c908a83e81c1" (UID: "5cfc6911-0010-445e-96b1-c908a83e81c1"). InnerVolumeSpecName "kube-api-access-xjfp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:12:08.092778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.092733    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/5cfc6911-0010-445e-96b1-c908a83e81c1-kube-api-access-xjfp2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:12:08.092778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.092772    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:12:08.092778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.092781    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfc6911-0010-445e-96b1-c908a83e81c1-webhook-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:12:08.654016 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.653979    2583 generic.go:356] "Generic (PLEG): container finished" podID="5cfc6911-0010-445e-96b1-c908a83e81c1" containerID="ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc" exitCode=0
Sep 15 12:12:08.654016 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.654017    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4"
Sep 15 12:12:08.654503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.654066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4" event={"ID":"5cfc6911-0010-445e-96b1-c908a83e81c1","Type":"ContainerDied","Data":"ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc"}
Sep 15 12:12:08.654503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.654124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76dbd9ddcd-t4wv4" event={"ID":"5cfc6911-0010-445e-96b1-c908a83e81c1","Type":"ContainerDied","Data":"fc31ba10fadb60eb0278826cb9830d6c999ec83f47e2e5c6b03fd81962dea45e"}
Sep 15 12:12:08.654503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.654148    2583 scope.go:117] "RemoveContainer" containerID="ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc"
Sep 15 12:12:08.663812 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.663791    2583 scope.go:117] "RemoveContainer" containerID="ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc"
Sep 15 12:12:08.664115 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:12:08.664092    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc\": container with ID starting with ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc not found: ID does not exist" containerID="ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc"
Sep 15 12:12:08.664195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.664123    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc"} err="failed to get container status \"ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc\": rpc error: code = NotFound desc = could not find container \"ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc\": container with ID starting with ec4f1fef53d5f1ad1b0c86278e60ae772bfe67034becd3cc606323e6ea7857fc not found: ID does not exist"
Sep 15 12:12:08.676781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.676747    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76dbd9ddcd-t4wv4"]
Sep 15 12:12:08.680071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:08.680047    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76dbd9ddcd-t4wv4"]
Sep 15 12:12:09.806739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:09.806700    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cfc6911-0010-445e-96b1-c908a83e81c1" path="/var/lib/kubelet/pods/5cfc6911-0010-445e-96b1-c908a83e81c1/volumes"
Sep 15 12:12:29.453679 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:29.453629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:30.817613 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.817580    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-865999984f-229wk"]
Sep 15 12:12:30.817979 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.817947    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cfc6911-0010-445e-96b1-c908a83e81c1" containerName="manager"
Sep 15 12:12:30.817979 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.817958    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cfc6911-0010-445e-96b1-c908a83e81c1" containerName="manager"
Sep 15 12:12:30.818071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.818050    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cfc6911-0010-445e-96b1-c908a83e81c1" containerName="manager"
Sep 15 12:12:30.823113 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.823093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.826037 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.825998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jhxcj\""
Sep 15 12:12:30.826174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.826002    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 12:12:30.826750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.826733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 12:12:30.835436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.835406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-865999984f-229wk"]
Sep 15 12:12:30.881575 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.881542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-metrics-certs\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.881575 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.881577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xv2\" (UniqueName: \"kubernetes.io/projected/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-kube-api-access-g9xv2\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.881780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.881608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-openshift-service-ca\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.881780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.881672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.982971 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.982939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-metrics-certs\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.983181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.982974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xv2\" (UniqueName: \"kubernetes.io/projected/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-kube-api-access-g9xv2\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.983181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.983078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-openshift-service-ca\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.983181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.983135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.983360 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:12:30.983271    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 15 12:12:30.983360 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:12:30.983356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls podName:fffb7b5b-85ff-4f02-919a-b49ca8a5b476 nodeName:}" failed. No retries permitted until 2026-09-15 12:12:31.483335163 +0000 UTC m=+632.179894734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls") pod "maas-api-865999984f-229wk" (UID: "fffb7b5b-85ff-4f02-919a-b49ca8a5b476") : secret "maas-api-serving-cert" not found
Sep 15 12:12:30.983990 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.983963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-openshift-service-ca\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.985878 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.985857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-metrics-certs\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:30.999507 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:30.999474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xv2\" (UniqueName: \"kubernetes.io/projected/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-kube-api-access-g9xv2\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:31.061810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.061777    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:31.225903 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.225820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:31.312795 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.312761    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:31.420168 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.420134    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"]
Sep 15 12:12:31.423508 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.423490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.428718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.428690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cphvm\""
Sep 15 12:12:31.429035 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.429002    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 12:12:31.436607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.436587    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"]
Sep 15 12:12:31.487378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.487295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:31.487378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.487366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2v2\" (UniqueName: \"kubernetes.io/projected/3dc0d539-2e6f-4b87-8ecb-d01c327ad983-kube-api-access-4f2v2\") pod \"payload-pre-processing-59fbc47bf8-26h95\" (UID: \"3dc0d539-2e6f-4b87-8ecb-d01c327ad983\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.487606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.487394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0d539-2e6f-4b87-8ecb-d01c327ad983-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-26h95\" (UID: \"3dc0d539-2e6f-4b87-8ecb-d01c327ad983\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.489869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.489832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls\") pod \"maas-api-865999984f-229wk\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") " pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:31.588374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.588340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2v2\" (UniqueName: \"kubernetes.io/projected/3dc0d539-2e6f-4b87-8ecb-d01c327ad983-kube-api-access-4f2v2\") pod \"payload-pre-processing-59fbc47bf8-26h95\" (UID: \"3dc0d539-2e6f-4b87-8ecb-d01c327ad983\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.588592 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.588381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0d539-2e6f-4b87-8ecb-d01c327ad983-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-26h95\" (UID: \"3dc0d539-2e6f-4b87-8ecb-d01c327ad983\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.589162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.589136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0d539-2e6f-4b87-8ecb-d01c327ad983-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-26h95\" (UID: \"3dc0d539-2e6f-4b87-8ecb-d01c327ad983\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.597511 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.597478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2v2\" (UniqueName: \"kubernetes.io/projected/3dc0d539-2e6f-4b87-8ecb-d01c327ad983-kube-api-access-4f2v2\") pod \"payload-pre-processing-59fbc47bf8-26h95\" (UID: \"3dc0d539-2e6f-4b87-8ecb-d01c327ad983\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.734579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.734544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:31.736692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:31.736662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:32.021631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:32.021598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"]
Sep 15 12:12:32.024220 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:12:32.024190    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc0d539_2e6f_4b87_8ecb_d01c327ad983.slice/crio-20cd59146e6d8e375e7123b866f4d17e39dd20fe5238f5b667e981e922f157d1 WatchSource:0}: Error finding container 20cd59146e6d8e375e7123b866f4d17e39dd20fe5238f5b667e981e922f157d1: Status 404 returned error can't find the container with id 20cd59146e6d8e375e7123b866f4d17e39dd20fe5238f5b667e981e922f157d1
Sep 15 12:12:32.041365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:32.041345    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-865999984f-229wk"]
Sep 15 12:12:32.044088 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:12:32.044058    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffb7b5b_85ff_4f02_919a_b49ca8a5b476.slice/crio-8244afef5c4cbe0f0507ad04bdce14508341c8eb432f478a506f3058217d5d20 WatchSource:0}: Error finding container 8244afef5c4cbe0f0507ad04bdce14508341c8eb432f478a506f3058217d5d20: Status 404 returned error can't find the container with id 8244afef5c4cbe0f0507ad04bdce14508341c8eb432f478a506f3058217d5d20
Sep 15 12:12:32.775149 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:32.775108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95" event={"ID":"3dc0d539-2e6f-4b87-8ecb-d01c327ad983","Type":"ContainerStarted","Data":"20cd59146e6d8e375e7123b866f4d17e39dd20fe5238f5b667e981e922f157d1"}
Sep 15 12:12:32.776357 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:32.776333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" event={"ID":"fffb7b5b-85ff-4f02-919a-b49ca8a5b476","Type":"ContainerStarted","Data":"8244afef5c4cbe0f0507ad04bdce14508341c8eb432f478a506f3058217d5d20"}
Sep 15 12:12:35.794356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:35.794321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95" event={"ID":"3dc0d539-2e6f-4b87-8ecb-d01c327ad983","Type":"ContainerStarted","Data":"ae27286e73d481c0da2ba7349ec0943a7919d5aae6b2b2747839342668e93793"}
Sep 15 12:12:35.795175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:35.794493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:35.795931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:35.795907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" event={"ID":"fffb7b5b-85ff-4f02-919a-b49ca8a5b476","Type":"ContainerStarted","Data":"80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773"}
Sep 15 12:12:35.796076 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:35.796060    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:35.811205 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:35.811155    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95" podStartSLOduration=1.814046717 podStartE2EDuration="4.811138961s" totalImagesPullingTime="2.997092244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:35.810271326 +0000 UTC m=+636.506830923" watchObservedRunningTime="2026-09-15 12:12:35.811138961 +0000 UTC m=+636.507698539"
Sep 15 12:12:35.831715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:35.831668    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" podStartSLOduration=2.904989445 podStartE2EDuration="5.831653742s" totalImagesPullingTime="2.926664297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:35.83015254 +0000 UTC m=+636.526712118" watchObservedRunningTime="2026-09-15 12:12:35.831653742 +0000 UTC m=+636.528213320"
Sep 15 12:12:41.807463 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:41.807432    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:12:42.814879 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:42.814822    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:42.912209 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:42.912177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:46.342804 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:46.342754    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:46.802827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:46.802747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-26h95"
Sep 15 12:12:54.103953 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.103913    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"]
Sep 15 12:12:54.108922 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.108900    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.111921 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.111893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 15 12:12:54.112044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.111896    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 12:12:54.112044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.111896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c9vjk\""
Sep 15 12:12:54.112044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.111927    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 12:12:54.116295 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.116270    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"]
Sep 15 12:12:54.189845 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.189810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.189845 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.189852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwnr\" (UniqueName: \"kubernetes.io/projected/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-kube-api-access-wjwnr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.190126 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.189956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.190126 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.189994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.190126 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.190098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.190270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.190141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.218617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.218563    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:54.291231 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291231 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291495 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291495 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwnr\" (UniqueName: \"kubernetes.io/projected/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-kube-api-access-wjwnr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.291859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.291839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.293586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.293564    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.293969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.293951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.300343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.300318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwnr\" (UniqueName: \"kubernetes.io/projected/af111aac-ecbe-4b67-a9f0-fb15a1b2c001-kube-api-access-wjwnr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rrd5t\" (UID: \"af111aac-ecbe-4b67-a9f0-fb15a1b2c001\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.420460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.420371    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:12:54.695202 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.695174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"]
Sep 15 12:12:54.697336 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:12:54.697310    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf111aac_ecbe_4b67_a9f0_fb15a1b2c001.slice/crio-9fce400d0ee5003a42f9099c4c4f9879f94fed2326a7573d0eca339e952d0cb9 WatchSource:0}: Error finding container 9fce400d0ee5003a42f9099c4c4f9879f94fed2326a7573d0eca339e952d0cb9: Status 404 returned error can't find the container with id 9fce400d0ee5003a42f9099c4c4f9879f94fed2326a7573d0eca339e952d0cb9
Sep 15 12:12:54.905686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:54.905639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t" event={"ID":"af111aac-ecbe-4b67-a9f0-fb15a1b2c001","Type":"ContainerStarted","Data":"9fce400d0ee5003a42f9099c4c4f9879f94fed2326a7573d0eca339e952d0cb9"}
Sep 15 12:12:55.912676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:55.912635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t" event={"ID":"af111aac-ecbe-4b67-a9f0-fb15a1b2c001","Type":"ContainerStarted","Data":"7867e52cce5a3bf0cf977b9e3de4d0b50b21faec4c6c48af5e10fb298e67cd6f"}
Sep 15 12:12:55.931344 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:55.931292    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t" podStartSLOduration=0.908878776 podStartE2EDuration="1.931275781s" totalImagesPullingTime="1.022397005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:55.929990119 +0000 UTC m=+656.626549714" watchObservedRunningTime="2026-09-15 12:12:55.931275781 +0000 UTC m=+656.627835358"
Sep 15 12:12:56.131449 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:56.131414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:12:57.625960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:12:57.625923    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:04.420787 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:04.420743    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:13:04.421233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:04.420802    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:13:04.433589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:04.433561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:13:04.972478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:04.972446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rrd5t"
Sep 15 12:13:19.097772 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:19.097736    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:19.196892 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:19.196854    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:21.131603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:21.131544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:29.154341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:29.154311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:29.241817 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:29.241782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:40.565951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:40.565868    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:13:55.832655 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:13:55.832556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:14:04.275537 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:14:04.275491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:14:04.659635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:14:04.659599    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:00.392515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.392471    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cbf5dfc84-9rs89"]
Sep 15 12:15:00.392973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.392700    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89" podUID="b404102f-059c-4575-b8b1-7fcd81b95d62" containerName="authorino" containerID="cri-o://097be6b89d8275f89618f92c08e51ca5ca6a5cfd1632db9ea2e0650b2461ff84" gracePeriod=30
Sep 15 12:15:00.549988 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.549952    2583 generic.go:356] "Generic (PLEG): container finished" podID="b404102f-059c-4575-b8b1-7fcd81b95d62" containerID="097be6b89d8275f89618f92c08e51ca5ca6a5cfd1632db9ea2e0650b2461ff84" exitCode=0
Sep 15 12:15:00.550162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.550038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89" event={"ID":"b404102f-059c-4575-b8b1-7fcd81b95d62","Type":"ContainerDied","Data":"097be6b89d8275f89618f92c08e51ca5ca6a5cfd1632db9ea2e0650b2461ff84"}
Sep 15 12:15:00.645858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.645790    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:15:00.788799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.788760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krq8c\" (UniqueName: \"kubernetes.io/projected/b404102f-059c-4575-b8b1-7fcd81b95d62-kube-api-access-krq8c\") pod \"b404102f-059c-4575-b8b1-7fcd81b95d62\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") "
Sep 15 12:15:00.788984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.788824    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b404102f-059c-4575-b8b1-7fcd81b95d62-tls-cert\") pod \"b404102f-059c-4575-b8b1-7fcd81b95d62\" (UID: \"b404102f-059c-4575-b8b1-7fcd81b95d62\") "
Sep 15 12:15:00.791053 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.791013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b404102f-059c-4575-b8b1-7fcd81b95d62-kube-api-access-krq8c" (OuterVolumeSpecName: "kube-api-access-krq8c") pod "b404102f-059c-4575-b8b1-7fcd81b95d62" (UID: "b404102f-059c-4575-b8b1-7fcd81b95d62"). InnerVolumeSpecName "kube-api-access-krq8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:15:00.799712 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.799684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b404102f-059c-4575-b8b1-7fcd81b95d62-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b404102f-059c-4575-b8b1-7fcd81b95d62" (UID: "b404102f-059c-4575-b8b1-7fcd81b95d62"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:15:00.890426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.890386    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krq8c\" (UniqueName: \"kubernetes.io/projected/b404102f-059c-4575-b8b1-7fcd81b95d62-kube-api-access-krq8c\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:15:00.890426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.890421    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b404102f-059c-4575-b8b1-7fcd81b95d62-tls-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:15:00.944687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:00.944597    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:01.051380 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.051347    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:01.557615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.557586    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89"
Sep 15 12:15:01.558063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.557589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cbf5dfc84-9rs89" event={"ID":"b404102f-059c-4575-b8b1-7fcd81b95d62","Type":"ContainerDied","Data":"d4c9469cbeee1deb2fd8a135db0c69eae03932f4c9117970645890ce941170d9"}
Sep 15 12:15:01.558063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.557713    2583 scope.go:117] "RemoveContainer" containerID="097be6b89d8275f89618f92c08e51ca5ca6a5cfd1632db9ea2e0650b2461ff84"
Sep 15 12:15:01.578605 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.578565    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cbf5dfc84-9rs89"]
Sep 15 12:15:01.583550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.583518    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cbf5dfc84-9rs89"]
Sep 15 12:15:01.807098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:01.807056    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b404102f-059c-4575-b8b1-7fcd81b95d62" path="/var/lib/kubelet/pods/b404102f-059c-4575-b8b1-7fcd81b95d62/volumes"
Sep 15 12:15:02.109551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:02.109507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:02.261179 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:02.261138    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:18.719575 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:18.719540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:18.822568 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:18.822527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:18.929233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:18.929201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:19.037228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.037157    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"]
Sep 15 12:15:19.037559 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.037547    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b404102f-059c-4575-b8b1-7fcd81b95d62" containerName="authorino"
Sep 15 12:15:19.037602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.037561    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b404102f-059c-4575-b8b1-7fcd81b95d62" containerName="authorino"
Sep 15 12:15:19.037636 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.037621    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b404102f-059c-4575-b8b1-7fcd81b95d62" containerName="authorino"
Sep 15 12:15:19.042151 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.042130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.044727 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.044700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9ba48f-openshift-default-dockercfg-dgnf9\""
Sep 15 12:15:19.044914 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.044901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 12:15:19.060125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.060100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"]
Sep 15 12:15:19.112532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.112498    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:19.143440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143403    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-credential-socket\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7202520-11ad-406a-b2a3-42c493494693-istiod-ca-cert\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-socket\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7202520-11ad-406a-b2a3-42c493494693-istio-podinfo\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-data\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-istio-token\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vtm2\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-kube-api-access-7vtm2\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-envoy\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.143951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.143759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-certs\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.204748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.204718    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:19.244830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.244800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7202520-11ad-406a-b2a3-42c493494693-istiod-ca-cert\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.244837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-socket\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.244865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7202520-11ad-406a-b2a3-42c493494693-istio-podinfo\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.244949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-data\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.244982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-istio-token\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vtm2\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-kube-api-access-7vtm2\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-envoy\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-certs\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-credential-socket\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245344 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-socket\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-credential-socket\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245563 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-certs\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245831 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-data\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.245958 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.245856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7202520-11ad-406a-b2a3-42c493494693-istiod-ca-cert\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.247350 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.247328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-envoy\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.247812 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.247793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7202520-11ad-406a-b2a3-42c493494693-istio-podinfo\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.253263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.253239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-istio-token\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.253511 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.253489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vtm2\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-kube-api-access-7vtm2\") pod \"e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") " pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.303914 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.303820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:19.352753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.352722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:19.403526 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.403378    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:19.628407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.628306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"]
Sep 15 12:15:19.630717 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:19.630688    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7202520_11ad_406a_b2a3_42c493494693.slice/crio-3d199247d48242473839e937e7731c7c20cad149d85e798c8658ec7e3d3b1044 WatchSource:0}: Error finding container 3d199247d48242473839e937e7731c7c20cad149d85e798c8658ec7e3d3b1044: Status 404 returned error can't find the container with id 3d199247d48242473839e937e7731c7c20cad149d85e798c8658ec7e3d3b1044
Sep 15 12:15:19.632918 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.632902    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:15:19.647208 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.647185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" event={"ID":"a7202520-11ad-406a-b2a3-42c493494693","Type":"ContainerStarted","Data":"3d199247d48242473839e937e7731c7c20cad149d85e798c8658ec7e3d3b1044"}
Sep 15 12:15:19.721295 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:19.721256    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:20.019907 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.019799    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:20.186478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.186436    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:20.268624 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.268592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:20.547352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.547323    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:20.634881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.634842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:20.731594 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.731553    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7950-0c044a"]
Sep 15 12:15:20.735315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.735291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:20.737800 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.737776    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q76n5\""
Sep 15 12:15:20.741922 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.741900    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7950-0c044a"]
Sep 15 12:15:20.758158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.758124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl29q\" (UniqueName: \"kubernetes.io/projected/55512135-f69a-4ff6-a446-cffed21d7709-kube-api-access-nl29q\") pod \"test-curl-7950-0c044a\" (UID: \"55512135-f69a-4ff6-a446-cffed21d7709\") " pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:20.858660 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.858622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl29q\" (UniqueName: \"kubernetes.io/projected/55512135-f69a-4ff6-a446-cffed21d7709-kube-api-access-nl29q\") pod \"test-curl-7950-0c044a\" (UID: \"55512135-f69a-4ff6-a446-cffed21d7709\") " pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:20.867874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:20.867785    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl29q\" (UniqueName: \"kubernetes.io/projected/55512135-f69a-4ff6-a446-cffed21d7709-kube-api-access-nl29q\") pod \"test-curl-7950-0c044a\" (UID: \"55512135-f69a-4ff6-a446-cffed21d7709\") " pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:21.049140 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:21.049100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:21.344130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:21.344095    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7950-0c044a"]
Sep 15 12:15:21.346460 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:21.346413    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55512135_f69a_4ff6_a446_cffed21d7709.slice/crio-6f8c3b87851e2f51f0e9f385565817b964ea9d67ec0df208f1c4e2f1e90c9e7e WatchSource:0}: Error finding container 6f8c3b87851e2f51f0e9f385565817b964ea9d67ec0df208f1c4e2f1e90c9e7e: Status 404 returned error can't find the container with id 6f8c3b87851e2f51f0e9f385565817b964ea9d67ec0df208f1c4e2f1e90c9e7e
Sep 15 12:15:21.667095 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:21.667052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7950-0c044a" event={"ID":"55512135-f69a-4ff6-a446-cffed21d7709","Type":"ContainerStarted","Data":"6f8c3b87851e2f51f0e9f385565817b964ea9d67ec0df208f1c4e2f1e90c9e7e"}
Sep 15 12:15:22.158389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:22.158352    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:22.251748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:22.251706    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:22.643279 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:22.643239    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:22.643387 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:22.643319    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:22.643387 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:22.643347    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:23.352298 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.352265    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"]
Sep 15 12:15:23.356070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.356047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.359704 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.359515    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4a9453-serving-cert\""
Sep 15 12:15:23.369381 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.369355    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"]
Sep 15 12:15:23.385516 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.385476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzht8\" (UniqueName: \"kubernetes.io/projected/5f961480-9129-4b78-a061-4c9ac71ae494-kube-api-access-vzht8\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.385688 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.385605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f961480-9129-4b78-a061-4c9ac71ae494-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.385688 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.385659    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.385805 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.385689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-metrics-certs\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.393885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.393850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:23.486331 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.486292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzht8\" (UniqueName: \"kubernetes.io/projected/5f961480-9129-4b78-a061-4c9ac71ae494-kube-api-access-vzht8\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.486542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.486381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f961480-9129-4b78-a061-4c9ac71ae494-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.486542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.486422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.486542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.486477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-metrics-certs\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.486542 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:23.486525    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-serving-cert: secret "maas-api-e2e-inf-a-4a9453-serving-cert" not found
Sep 15 12:15:23.486737 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:23.486588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls podName:5f961480-9129-4b78-a061-4c9ac71ae494 nodeName:}" failed. No retries permitted until 2026-09-15 12:15:23.986565638 +0000 UTC m=+804.683125202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls") pod "maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" (UID: "5f961480-9129-4b78-a061-4c9ac71ae494") : secret "maas-api-e2e-inf-a-4a9453-serving-cert" not found
Sep 15 12:15:23.487298 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.487281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f961480-9129-4b78-a061-4c9ac71ae494-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.488926 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.488904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-metrics-certs\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.503786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.503755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzht8\" (UniqueName: \"kubernetes.io/projected/5f961480-9129-4b78-a061-4c9ac71ae494-kube-api-access-vzht8\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.504365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.504310    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:23.593832 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.593800    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:23.678830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.678733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" event={"ID":"a7202520-11ad-406a-b2a3-42c493494693","Type":"ContainerStarted","Data":"a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"}
Sep 15 12:15:23.698010 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.697942    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podStartSLOduration=1.688105083 podStartE2EDuration="4.69792133s" totalImagesPullingTime="3.009816247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:23.697194885 +0000 UTC m=+804.393754485" watchObservedRunningTime="2026-09-15 12:15:23.69792133 +0000 UTC m=+804.394480908"
Sep 15 12:15:23.786449 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.786412    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:23.990840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.990683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:23.994139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:23.994103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls\") pod \"maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:24.024802 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.024759    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:24.131433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.131395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:24.177061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.177010    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"]
Sep 15 12:15:24.180875 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.180848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.183608 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.183580    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4a9453-dockercfg-8wlbx\""
Sep 15 12:15:24.183869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.183848    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4a9453\""
Sep 15 12:15:24.190413 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.190387    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"]
Sep 15 12:15:24.269115 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.268966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:24.294805 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.294769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.294951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.294882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjqhs\" (UniqueName: \"kubernetes.io/projected/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-kube-api-access-pjqhs\") pod \"payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.353518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.353426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:24.360280 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.360200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:24.396014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.395977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjqhs\" (UniqueName: \"kubernetes.io/projected/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-kube-api-access-pjqhs\") pod \"payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.396252 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.396101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.397144 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.397096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.405347 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.405175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjqhs\" (UniqueName: \"kubernetes.io/projected/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-kube-api-access-pjqhs\") pod \"payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.495785 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.495738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:24.561680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.561590    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"]
Sep 15 12:15:24.685089 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.685014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:24.686160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.686141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:24.779282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:24.779253    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"]
Sep 15 12:15:25.199582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.199550    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"]
Sep 15 12:15:25.222578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.222543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"]
Sep 15 12:15:25.222749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.222669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.225813 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.225783    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-680229-serving-cert\""
Sep 15 12:15:25.263985 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.263943    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:25.266916 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:25.266883    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f961480_9129_4b78_a061_4c9ac71ae494.slice/crio-2da2faf1d421f04d0bd1f475919d0671c0e5f91ea586d35d20377c75d4ed41a3 WatchSource:0}: Error finding container 2da2faf1d421f04d0bd1f475919d0671c0e5f91ea586d35d20377c75d4ed41a3: Status 404 returned error can't find the container with id 2da2faf1d421f04d0bd1f475919d0671c0e5f91ea586d35d20377c75d4ed41a3
Sep 15 12:15:25.267564 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:25.267529    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41785fc0_b7e9_40d1_8f42_4ceaa01bd367.slice/crio-08c270a6a3a57866a9a92d0c8ed159646537716f16fc4afda817fae0be5076a2 WatchSource:0}: Error finding container 08c270a6a3a57866a9a92d0c8ed159646537716f16fc4afda817fae0be5076a2: Status 404 returned error can't find the container with id 08c270a6a3a57866a9a92d0c8ed159646537716f16fc4afda817fae0be5076a2
Sep 15 12:15:25.305675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.305622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-metrics-certs\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.305842 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.305723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a5da10-3ddd-4880-8d47-001c454ca14a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.305842 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.305818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.306008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.305890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmln\" (UniqueName: \"kubernetes.io/projected/83a5da10-3ddd-4880-8d47-001c454ca14a-kube-api-access-pxmln\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.407270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.407237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a5da10-3ddd-4880-8d47-001c454ca14a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.407670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.407308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.407670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.407363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmln\" (UniqueName: \"kubernetes.io/projected/83a5da10-3ddd-4880-8d47-001c454ca14a-kube-api-access-pxmln\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.407670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.407417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-metrics-certs\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.408821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.408787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a5da10-3ddd-4880-8d47-001c454ca14a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.410938 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.410882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.410938 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.410884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-metrics-certs\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.411960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.411930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:25.419971 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.419947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmln\" (UniqueName: \"kubernetes.io/projected/83a5da10-3ddd-4880-8d47-001c454ca14a-kube-api-access-pxmln\") pod \"maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.511789 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.511702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:25.536769 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.536728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:25.692190 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.692150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" event={"ID":"41785fc0-b7e9-40d1-8f42-4ceaa01bd367","Type":"ContainerStarted","Data":"7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0"}
Sep 15 12:15:25.692390 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.692257    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:25.692390 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.692282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" event={"ID":"41785fc0-b7e9-40d1-8f42-4ceaa01bd367","Type":"ContainerStarted","Data":"08c270a6a3a57866a9a92d0c8ed159646537716f16fc4afda817fae0be5076a2"}
Sep 15 12:15:25.693738 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.693698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7950-0c044a" event={"ID":"55512135-f69a-4ff6-a446-cffed21d7709","Type":"ContainerStarted","Data":"4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7"}
Sep 15 12:15:25.694952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.694925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" event={"ID":"5f961480-9129-4b78-a061-4c9ac71ae494","Type":"ContainerStarted","Data":"2da2faf1d421f04d0bd1f475919d0671c0e5f91ea586d35d20377c75d4ed41a3"}
Sep 15 12:15:25.713133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.713052    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" podStartSLOduration=1.7130189900000001 podStartE2EDuration="1.71301899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:25.709000025 +0000 UTC m=+806.405559634" watchObservedRunningTime="2026-09-15 12:15:25.71301899 +0000 UTC m=+806.409578570"
Sep 15 12:15:25.728646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.728600    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7950-0c044a" podStartSLOduration=1.738359013 podStartE2EDuration="5.728583713s" totalImagesPullingTime="3.9902247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:25.726589764 +0000 UTC m=+806.423149342" watchObservedRunningTime="2026-09-15 12:15:25.728583713 +0000 UTC m=+806.425143291"
Sep 15 12:15:25.817766 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.817719    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"]
Sep 15 12:15:25.823767 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:25.823732    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a5da10_3ddd_4880_8d47_001c454ca14a.slice/crio-ea8e799510ec9301c1d83371dc58ebac3c195e8c6e61879a93744fdb3ce8bc1f WatchSource:0}: Error finding container ea8e799510ec9301c1d83371dc58ebac3c195e8c6e61879a93744fdb3ce8bc1f: Status 404 returned error can't find the container with id ea8e799510ec9301c1d83371dc58ebac3c195e8c6e61879a93744fdb3ce8bc1f
Sep 15 12:15:25.832004 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.831917    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:25.953478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:25.953396    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:26.043187 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.043089    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:26.399001 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.398966    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7950-0c044a"]
Sep 15 12:15:26.402758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.402727    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7950-0c044a"]
Sep 15 12:15:26.566591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.566557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:26.702314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.702210    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" event={"ID":"5f961480-9129-4b78-a061-4c9ac71ae494","Type":"ContainerStarted","Data":"7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074"}
Sep 15 12:15:26.702314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.702286    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:26.703993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.703962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn" event={"ID":"83a5da10-3ddd-4880-8d47-001c454ca14a","Type":"ContainerStarted","Data":"a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2"}
Sep 15 12:15:26.704119 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.704015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn" event={"ID":"83a5da10-3ddd-4880-8d47-001c454ca14a","Type":"ContainerStarted","Data":"ea8e799510ec9301c1d83371dc58ebac3c195e8c6e61879a93744fdb3ce8bc1f"}
Sep 15 12:15:26.704195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.704173    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:26.704258 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:26.704219    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7950-0c044a" container="test-curl-7950-0c044a"
Sep 15 12:15:26.730133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.730080    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" podStartSLOduration=3.28496416 podStartE2EDuration="3.730061795s" totalImagesPullingTime="445.097635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:26.728694821 +0000 UTC m=+807.425254424" watchObservedRunningTime="2026-09-15 12:15:26.730061795 +0000 UTC m=+807.426621373"
Sep 15 12:15:26.750095 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:26.750015    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn" podStartSLOduration=1.387810112 podStartE2EDuration="1.749996012s" totalImagesPullingTime="362.1859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:26.748135027 +0000 UTC m=+807.444694625" watchObservedRunningTime="2026-09-15 12:15:26.749996012 +0000 UTC m=+807.446555594"
Sep 15 12:15:27.437329 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:27.437294    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:27.710301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:27.710190    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7950-0c044a" podUID="55512135-f69a-4ff6-a446-cffed21d7709" containerName="test-curl-7950-0c044a" containerID="cri-o://4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7" gracePeriod=2
Sep 15 12:15:27.712247 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:27.712212    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7950-0c044a\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="55512135-f69a-4ff6-a446-cffed21d7709" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:28.350097 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.350062    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:28.379896 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.379863    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"]
Sep 15 12:15:28.380456 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.380430    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55512135-f69a-4ff6-a446-cffed21d7709" containerName="test-curl-7950-0c044a"
Sep 15 12:15:28.380456 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.380457    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="55512135-f69a-4ff6-a446-cffed21d7709" containerName="test-curl-7950-0c044a"
Sep 15 12:15:28.380666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.380589    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="55512135-f69a-4ff6-a446-cffed21d7709" containerName="test-curl-7950-0c044a"
Sep 15 12:15:28.384616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.384595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.387088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.387063    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9ba48f-serving-cert\""
Sep 15 12:15:28.398855 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.398831    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"]
Sep 15 12:15:28.425603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.425566    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:28.439511 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.439482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.439641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.439575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db24a8af-e6f2-4246-8654-267122705fb6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.439700 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.439646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpz4\" (UniqueName: \"kubernetes.io/projected/db24a8af-e6f2-4246-8654-267122705fb6-kube-api-access-pfpz4\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.439743 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.439702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-metrics-certs\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.504397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.504361    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:28.540453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.540419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.540641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.540476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db24a8af-e6f2-4246-8654-267122705fb6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.540641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.540508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpz4\" (UniqueName: \"kubernetes.io/projected/db24a8af-e6f2-4246-8654-267122705fb6-kube-api-access-pfpz4\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.540641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.540544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-metrics-certs\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.540641 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:28.540591    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-serving-cert: secret "maas-api-e2e-worker-w4-9ba48f-serving-cert" not found
Sep 15 12:15:28.540859 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:28.540671    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls podName:db24a8af-e6f2-4246-8654-267122705fb6 nodeName:}" failed. No retries permitted until 2026-09-15 12:15:29.040648795 +0000 UTC m=+809.737208365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls") pod "maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" (UID: "db24a8af-e6f2-4246-8654-267122705fb6") : secret "maas-api-e2e-worker-w4-9ba48f-serving-cert" not found
Sep 15 12:15:28.541205 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.541179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db24a8af-e6f2-4246-8654-267122705fb6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.543074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.543042    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-metrics-certs\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.552510 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.552485    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpz4\" (UniqueName: \"kubernetes.io/projected/db24a8af-e6f2-4246-8654-267122705fb6-kube-api-access-pfpz4\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:28.604226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.604140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:28.926473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:28.926365    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:29.012511 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.012453    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:29.046126 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.046089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:29.048733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.048702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:29.121977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.121937    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:29.215037 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.214929    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:29.296468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.296433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:29.427479 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.427438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:29.575755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.575729    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"]
Sep 15 12:15:29.577820 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:29.577793    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb24a8af_e6f2_4246_8654_267122705fb6.slice/crio-763a0fb4752188b9f45a3d48498d0b47449d26321a6f1aece7de89583d5482f8 WatchSource:0}: Error finding container 763a0fb4752188b9f45a3d48498d0b47449d26321a6f1aece7de89583d5482f8: Status 404 returned error can't find the container with id 763a0fb4752188b9f45a3d48498d0b47449d26321a6f1aece7de89583d5482f8
Sep 15 12:15:29.723105 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.723070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" event={"ID":"db24a8af-e6f2-4246-8654-267122705fb6","Type":"ContainerStarted","Data":"763a0fb4752188b9f45a3d48498d0b47449d26321a6f1aece7de89583d5482f8"}
Sep 15 12:15:29.870403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.870372    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:29.872228 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:29.872204    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7950-0c044a\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="55512135-f69a-4ff6-a446-cffed21d7709" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:29.956422 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.956379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl29q\" (UniqueName: \"kubernetes.io/projected/55512135-f69a-4ff6-a446-cffed21d7709-kube-api-access-nl29q\") pod \"55512135-f69a-4ff6-a446-cffed21d7709\" (UID: \"55512135-f69a-4ff6-a446-cffed21d7709\") "
Sep 15 12:15:29.958885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:29.958854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55512135-f69a-4ff6-a446-cffed21d7709-kube-api-access-nl29q" (OuterVolumeSpecName: "kube-api-access-nl29q") pod "55512135-f69a-4ff6-a446-cffed21d7709" (UID: "55512135-f69a-4ff6-a446-cffed21d7709"). InnerVolumeSpecName "kube-api-access-nl29q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:15:30.057281 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.057244    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl29q\" (UniqueName: \"kubernetes.io/projected/55512135-f69a-4ff6-a446-cffed21d7709-kube-api-access-nl29q\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:15:30.290917 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.290828    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:30.703585 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.703545    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:30.729852 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.729811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" event={"ID":"db24a8af-e6f2-4246-8654-267122705fb6","Type":"ContainerStarted","Data":"5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4"}
Sep 15 12:15:30.730086 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.729890    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:30.731129 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.731103    2583 generic.go:356] "Generic (PLEG): container finished" podID="55512135-f69a-4ff6-a446-cffed21d7709" containerID="4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7" exitCode=137
Sep 15 12:15:30.731230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.731140    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:30.731230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.731195    2583 scope.go:117] "RemoveContainer" containerID="4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7"
Sep 15 12:15:30.731995 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:30.731971    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7950-0c044a\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="55512135-f69a-4ff6-a446-cffed21d7709" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:30.740720 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.740697    2583 scope.go:117] "RemoveContainer" containerID="4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7"
Sep 15 12:15:30.741050 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:30.741015    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7\": container with ID starting with 4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7 not found: ID does not exist" containerID="4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7"
Sep 15 12:15:30.741122 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.741061    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7"} err="failed to get container status \"4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7\": rpc error: code = NotFound desc = could not find container \"4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7\": container with ID starting with 4e2f7c4877c51ae13f15a45a29508b37935b2b09a7d4972694ae704756a7dcd7 not found: ID does not exist"
Sep 15 12:15:30.755412 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:30.755328    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7950-0c044a\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="55512135-f69a-4ff6-a446-cffed21d7709" pod="opendatahub/test-curl-7950-0c044a"
Sep 15 12:15:30.757216 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:30.757175    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" podStartSLOduration=2.356795428 podStartE2EDuration="2.757158494s" totalImagesPullingTime="400.363066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:30.753341513 +0000 UTC m=+811.449901132" watchObservedRunningTime="2026-09-15 12:15:30.757158494 +0000 UTC m=+811.453718071"
Sep 15 12:15:31.024105 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:31.023994    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:31.241627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:31.241592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:31.807761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:31.807722    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55512135-f69a-4ff6-a446-cffed21d7709" path="/var/lib/kubelet/pods/55512135-f69a-4ff6-a446-cffed21d7709/volumes"
Sep 15 12:15:32.716518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:32.716483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:15:32.716997 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:32.716773    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:15:34.204830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:34.204794    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:34.919783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:34.919745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:35.005378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:35.005338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:35.335816 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:35.335782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:35.502887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:35.502853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:35.614756 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:35.614667    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:36.401209 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:36.401172    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:36.706583 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:36.706498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:15:36.746251 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:36.746220    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:15:37.305674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:37.305640    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:37.931314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:37.931282    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:38.233362 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.233259    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"]
Sep 15 12:15:38.239189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.239136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.242213 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.242187    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-0e10f0-serving-cert\""
Sep 15 12:15:38.251145 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.251116    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"]
Sep 15 12:15:38.297102 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.297058    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:38.333123 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.333063    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-metrics-certs\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.333304 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.333186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx88x\" (UniqueName: \"kubernetes.io/projected/65a614b8-6ac0-4a39-9642-a28dad3159d5-kube-api-access-zx88x\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.333377 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.333303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a614b8-6ac0-4a39-9642-a28dad3159d5-openshift-service-ca\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.333377 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.333341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-maas-api-tls\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.427973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.427930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:38.434665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.434616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-metrics-certs\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.434869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.434688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx88x\" (UniqueName: \"kubernetes.io/projected/65a614b8-6ac0-4a39-9642-a28dad3159d5-kube-api-access-zx88x\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.434869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.434777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a614b8-6ac0-4a39-9642-a28dad3159d5-openshift-service-ca\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.434869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.434812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-maas-api-tls\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.436856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.436817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a614b8-6ac0-4a39-9642-a28dad3159d5-openshift-service-ca\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.440921 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.440883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-metrics-certs\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.447657 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.447623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-maas-api-tls\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.452125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.452091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx88x\" (UniqueName: \"kubernetes.io/projected/65a614b8-6ac0-4a39-9642-a28dad3159d5-kube-api-access-zx88x\") pod \"maas-api-e2e-autores-0e10f0-965c475df-drxgr\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.526106 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.525599    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:38.553755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.553698    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:38.856685 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:38.856658    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"]
Sep 15 12:15:38.859428 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:38.859401    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a614b8_6ac0_4a39_9642_a28dad3159d5.slice/crio-982a8692096238211d00f17688808bd3fccd8aade727b7beb315d1aff15c42bc WatchSource:0}: Error finding container 982a8692096238211d00f17688808bd3fccd8aade727b7beb315d1aff15c42bc: Status 404 returned error can't find the container with id 982a8692096238211d00f17688808bd3fccd8aade727b7beb315d1aff15c42bc
Sep 15 12:15:39.087063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.086948    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.179724 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.179643    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.486228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.486147    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"]
Sep 15 12:15:39.491228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.491204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.493498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.493478    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7d7eff3e-serving-cert\""
Sep 15 12:15:39.502960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.502579    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"]
Sep 15 12:15:39.532978 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.532933    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.595576 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.595522    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.651669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.651629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-metrics-certs\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.651669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.651670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-openshift-service-ca\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.651883 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.651714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-maas-api-tls\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.651883 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.651815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hl9j\" (UniqueName: \"kubernetes.io/projected/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-kube-api-access-2hl9j\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.696404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.696364    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.753337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.753252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-maas-api-tls\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.753627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.753601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hl9j\" (UniqueName: \"kubernetes.io/projected/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-kube-api-access-2hl9j\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.753936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.753912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-metrics-certs\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.754130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.754108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-openshift-service-ca\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.754735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.754711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-openshift-service-ca\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.756352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.756323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-maas-api-tls\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.756520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.756503    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-metrics-certs\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.772387 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.772359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hl9j\" (UniqueName: \"kubernetes.io/projected/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-kube-api-access-2hl9j\") pod \"maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.785749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.785714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr" event={"ID":"65a614b8-6ac0-4a39-9642-a28dad3159d5","Type":"ContainerStarted","Data":"982a8692096238211d00f17688808bd3fccd8aade727b7beb315d1aff15c42bc"}
Sep 15 12:15:39.796418 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.796381    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.807346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.807313    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:39.895989 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.895848    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:39.995577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:39.995543    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:40.107578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.106930    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"]
Sep 15 12:15:40.112612 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:40.112549    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8817e72b_cff4_40ad_96f5_b2ec26b2c5cc.slice/crio-a3ba3f655c64e0c55e21bc03fc0fdc81f969352eefab9f5b93a3d2605c499d7c WatchSource:0}: Error finding container a3ba3f655c64e0c55e21bc03fc0fdc81f969352eefab9f5b93a3d2605c499d7c: Status 404 returned error can't find the container with id a3ba3f655c64e0c55e21bc03fc0fdc81f969352eefab9f5b93a3d2605c499d7c
Sep 15 12:15:40.138080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.137967    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:40.235470 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.235192    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:40.665139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.665092    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:40.792899 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.792812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr" event={"ID":"65a614b8-6ac0-4a39-9642-a28dad3159d5","Type":"ContainerStarted","Data":"e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5"}
Sep 15 12:15:40.793061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.793045    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:40.794495 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.794465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r" event={"ID":"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc","Type":"ContainerStarted","Data":"da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd"}
Sep 15 12:15:40.794618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.794504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r" event={"ID":"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc","Type":"ContainerStarted","Data":"a3ba3f655c64e0c55e21bc03fc0fdc81f969352eefab9f5b93a3d2605c499d7c"}
Sep 15 12:15:40.794618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.794548    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:40.817082 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:40.816997    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr" podStartSLOduration=1.206355049 podStartE2EDuration="2.816972003s" totalImagesPullingTime="1.610616954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:40.812717948 +0000 UTC m=+821.509277553" watchObservedRunningTime="2026-09-15 12:15:40.816972003 +0000 UTC m=+821.513531582"
Sep 15 12:15:41.592439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:41.592398    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:41.992182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:41.992087    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:42.332805 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.332766    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:42.547890 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.547854    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:42.649644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.636821    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r" podStartSLOduration=3.29256641 podStartE2EDuration="3.636795334s" totalImagesPullingTime="344.228924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:40.834870118 +0000 UTC m=+821.531429717" watchObservedRunningTime="2026-09-15 12:15:42.636795334 +0000 UTC m=+823.333354914"
Sep 15 12:15:42.649644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.645373    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"]
Sep 15 12:15:42.652737 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.651796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.658200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.657996    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-69d7ba-openshift-default-dockercfg-72jn7\""
Sep 15 12:15:42.668322 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.668275    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"]
Sep 15 12:15:42.669425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.669328    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:42.776553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.776519    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:42.787189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-credential-socket\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-token\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-data\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-podinfo\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dde6444-40e8-433a-8f97-3759c48d9c23-istiod-ca-cert\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-certs\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-socket\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-envoy\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.787565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.787541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtt6m\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-kube-api-access-dtt6m\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.883408 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.883368    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:42.888683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-socket\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.888786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-envoy\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.888786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt6m\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-kube-api-access-dtt6m\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.888888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-credential-socket\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.888888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-token\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-data\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-podinfo\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.888993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dde6444-40e8-433a-8f97-3759c48d9c23-istiod-ca-cert\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889191 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.889054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-certs\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889191 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.889147    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-socket\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.889520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-data\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.889568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-credential-socket\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889891 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.889869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dde6444-40e8-433a-8f97-3759c48d9c23-istiod-ca-cert\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.889958 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.889876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-certs\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.891144 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.891124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-envoy\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.891829 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.891807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-podinfo\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.897382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.897364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-token\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.897766 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.897750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt6m\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-kube-api-access-dtt6m\") pod \"e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") " pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:42.967632 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:42.967534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:43.274274 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.274241    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"]
Sep 15 12:15:43.277164 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:43.277131    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dde6444_40e8_433a_8f97_3759c48d9c23.slice/crio-8b3e76861ecbd4c73820c2e3963fb5b55a344229493c02fc6a7de360d6502dd9 WatchSource:0}: Error finding container 8b3e76861ecbd4c73820c2e3963fb5b55a344229493c02fc6a7de360d6502dd9: Status 404 returned error can't find the container with id 8b3e76861ecbd4c73820c2e3963fb5b55a344229493c02fc6a7de360d6502dd9
Sep 15 12:15:43.279624 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.279593    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:43.279721 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.279654    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:43.279721 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.279687    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:43.499920 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.499881    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:43.827087 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.827037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" event={"ID":"2dde6444-40e8-433a-8f97-3759c48d9c23","Type":"ContainerStarted","Data":"cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"}
Sep 15 12:15:43.827550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.827117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" event={"ID":"2dde6444-40e8-433a-8f97-3759c48d9c23","Type":"ContainerStarted","Data":"8b3e76861ecbd4c73820c2e3963fb5b55a344229493c02fc6a7de360d6502dd9"}
Sep 15 12:15:43.854763 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.854702    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podStartSLOduration=1.8546803120000002 podStartE2EDuration="1.854680312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:43.850308791 +0000 UTC m=+824.546868380" watchObservedRunningTime="2026-09-15 12:15:43.854680312 +0000 UTC m=+824.551239891"
Sep 15 12:15:43.968133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:43.968098    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:44.435592 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:44.435556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:44.531070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:44.531006    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:44.631786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:44.631744    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:44.974773 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:44.974737    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:45.522464 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:45.522428    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:45.840699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:45.840663    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:45.841658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:45.841628    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:15:46.512773 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:46.512734    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:46.619587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:46.619544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:46.806145 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:46.806050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:15:46.806695 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:46.806666    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:15:47.043574 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.043531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:47.262827 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:47.262790    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7202520_11ad_406a_b2a3_42c493494693.slice/crio-conmon-a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 12:15:47.853973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.853942    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7202520-11ad-406a-b2a3-42c493494693" containerID="a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4" exitCode=0
Sep 15 12:15:47.854417 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.854017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" event={"ID":"a7202520-11ad-406a-b2a3-42c493494693","Type":"ContainerDied","Data":"a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"}
Sep 15 12:15:47.854504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.854490    2583 scope.go:117] "RemoveContainer" containerID="a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"
Sep 15 12:15:47.855668 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.855642    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:47.855735 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.855706    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:47.855771 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.855739    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:15:47.933928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:47.933896    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:48.602859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.602824    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"]
Sep 15 12:15:48.605652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.605590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.609436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.609415    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-69d7ba-serving-cert\""
Sep 15 12:15:48.626050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.625901    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"]
Sep 15 12:15:48.639478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.639441    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:48.747094 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.747043    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:48.747332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.747185    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-openshift-service-ca\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.747332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.747222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.747332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.747245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-metrics-certs\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.747501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.747433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgpv\" (UniqueName: \"kubernetes.io/projected/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-kube-api-access-nrgpv\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.848005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.847967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.848005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.848006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-metrics-certs\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.848280 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:48.848138    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-serving-cert: secret "maas-api-e2e-inf-b-69d7ba-serving-cert" not found
Sep 15 12:15:48.848280 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:15:48.848212    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls podName:aaa847bb-1f34-4b2e-8540-ec63c5c8e192 nodeName:}" failed. No retries permitted until 2026-09-15 12:15:49.348196363 +0000 UTC m=+830.044755919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls") pod "maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" (UID: "aaa847bb-1f34-4b2e-8540-ec63c5c8e192") : secret "maas-api-e2e-inf-b-69d7ba-serving-cert" not found
Sep 15 12:15:48.848280 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.848142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgpv\" (UniqueName: \"kubernetes.io/projected/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-kube-api-access-nrgpv\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.848471 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.848317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-openshift-service-ca\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.848913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.848894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-openshift-service-ca\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.850664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.850639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-metrics-certs\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.862426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.862356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" event={"ID":"a7202520-11ad-406a-b2a3-42c493494693","Type":"ContainerStarted","Data":"51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2"}
Sep 15 12:15:48.864302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.864277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgpv\" (UniqueName: \"kubernetes.io/projected/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-kube-api-access-nrgpv\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:48.924005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:48.923963    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:49.147606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.147528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:49.240212 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.240178    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:49.353883 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.353841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:49.354228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.354075    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 15 12:15:49.354228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.354154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:49.354228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.354146    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 15 12:15:49.356684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.356665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls\") pod \"maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:49.533296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.533185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:49.775759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.775724    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:49.813885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.810170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"]
Sep 15 12:15:49.825889 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:15:49.824588    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa847bb_1f34_4b2e_8540_ec63c5c8e192.slice/crio-e18a4732fadd334e533b25cd0e90bc50e9395b8819dd207d57791259cb7863b2 WatchSource:0}: Error finding container e18a4732fadd334e533b25cd0e90bc50e9395b8819dd207d57791259cb7863b2: Status 404 returned error can't find the container with id e18a4732fadd334e533b25cd0e90bc50e9395b8819dd207d57791259cb7863b2
Sep 15 12:15:49.869532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:49.869498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" event={"ID":"aaa847bb-1f34-4b2e-8540-ec63c5c8e192","Type":"ContainerStarted","Data":"e18a4732fadd334e533b25cd0e90bc50e9395b8819dd207d57791259cb7863b2"}
Sep 15 12:15:50.353947 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:50.353909    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 15 12:15:50.354189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:50.353965    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 15 12:15:50.683342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:50.683300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:50.876846 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:50.876807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" event={"ID":"aaa847bb-1f34-4b2e-8540-ec63c5c8e192","Type":"ContainerStarted","Data":"0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d"}
Sep 15 12:15:50.877289 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:50.876874    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:50.898481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:50.898426    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" podStartSLOduration=2.5920067639999997 podStartE2EDuration="2.898410785s" totalImagesPullingTime="306.404021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:15:50.895642119 +0000 UTC m=+831.592201698" watchObservedRunningTime="2026-09-15 12:15:50.898410785 +0000 UTC m=+831.594970423"
Sep 15 12:15:51.353951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:51.353911    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 15 12:15:51.354284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:51.353992    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 15 12:15:51.620444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:51.620343    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:52.274298 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:52.274251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:52.353457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:52.353419    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 15 12:15:52.353623 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:52.353477    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 15 12:15:52.373688 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:52.373655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:52.690716 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:52.690676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:53.353436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:53.353393    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 15 12:15:53.353849 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:53.353462    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 15 12:15:53.478525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:53.478496    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:54.232754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:54.232707    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:54.353581 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:54.353538    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 15 12:15:54.353982 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:54.353612    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 15 12:15:54.685644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:54.685594    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:55.214290 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:55.214254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:55.320673 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:55.320631    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:55.358272 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:55.358243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:55.359296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:55.359275    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:15:56.002582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:56.002550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:56.691760 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:56.691718    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:56.887670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:56.887634    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:15:57.415861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:57.415829    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:58.126439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:58.126403    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:58.692439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:58.692394    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:58.790586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:58.790534    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:58.892281 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:58.892245    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:15:59.907582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:15:59.907531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:00.915728 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:00.915687    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:01.775237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:01.775199    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:02.536477 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:02.536446    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:03.294668 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:03.294635    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:04.115335 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:04.115292    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:04.887133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:04.887094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:05.635338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:05.635302    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:05.760665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:05.760627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:06.651499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:06.651463    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:09.604573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:09.604532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:10.992017 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:10.991926    2583 generic.go:356] "Generic (PLEG): container finished" podID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerID="cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3" exitCode=0
Sep 15 12:16:10.992017 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:10.991999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" event={"ID":"2dde6444-40e8-433a-8f97-3759c48d9c23","Type":"ContainerDied","Data":"cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"}
Sep 15 12:16:10.992490 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:10.992438    2583 scope.go:117] "RemoveContainer" containerID="cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"
Sep 15 12:16:10.993654 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:10.993628    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:10.993709 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:10.993699    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:10.993744 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:10.993728    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:11.999696 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:11.999664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" event={"ID":"2dde6444-40e8-433a-8f97-3759c48d9c23","Type":"ContainerStarted","Data":"87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb"}
Sep 15 12:16:12.968782 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:12.968740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:16:12.970095 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:12.970066    2583 patch_prober.go:28] interesting pod/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 12:16:12.970233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:12.970118    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 12:16:13.968398 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:13.968364    2583 patch_prober.go:28] interesting pod/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 12:16:13.968754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:13.968420    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 12:16:14.831048 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:14.830984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:14.968847 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:14.968811    2583 patch_prober.go:28] interesting pod/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 12:16:14.969233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:14.968876    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 12:16:15.841600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:15.841557    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:16:15.968178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:15.968143    2583 patch_prober.go:28] interesting pod/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 12:16:15.968345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:15.968194    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 12:16:16.410896 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:16.410862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:16.968746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:16.968706    2583 patch_prober.go:28] interesting pod/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 12:16:16.968947 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:16.968767    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 12:16:17.015529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.015476    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"]
Sep 15 12:16:17.016618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.016577    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r" podUID="8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" containerName="maas-api" containerID="cri-o://da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd" gracePeriod=30
Sep 15 12:16:17.054204 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.054171    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.154711 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.154673    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.252135 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.252054    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.277044 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.277009    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:16:17.351611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.351579    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.354513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.354489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-maas-api-tls\") pod \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") "
Sep 15 12:16:17.354603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.354559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hl9j\" (UniqueName: \"kubernetes.io/projected/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-kube-api-access-2hl9j\") pod \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") "
Sep 15 12:16:17.354642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.354613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-openshift-service-ca\") pod \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") "
Sep 15 12:16:17.354677 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.354659    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-metrics-certs\") pod \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\" (UID: \"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc\") "
Sep 15 12:16:17.355063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.355011    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" (UID: "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:17.356896 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.356875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" (UID: "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:16:17.357149 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.357114    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" (UID: "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:16:17.357239 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.357164    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-kube-api-access-2hl9j" (OuterVolumeSpecName: "kube-api-access-2hl9j") pod "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" (UID: "8817e72b-cff4-40ad-96f5-b2ec26b2c5cc"). InnerVolumeSpecName "kube-api-access-2hl9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:17.451532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.451495    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.455699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.455675    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:17.455699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.455701    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:17.455835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.455711    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:17.455835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.455720    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hl9j\" (UniqueName: \"kubernetes.io/projected/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc-kube-api-access-2hl9j\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:17.679134 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.679100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.958942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.958853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:17.972490 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.972467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:16:17.973349 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:17.973331    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:16:18.052245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.052212    2583 generic.go:356] "Generic (PLEG): container finished" podID="8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" containerID="da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd" exitCode=0
Sep 15 12:16:18.052420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.052252    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"
Sep 15 12:16:18.052420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.052299    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r" event={"ID":"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc","Type":"ContainerDied","Data":"da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd"}
Sep 15 12:16:18.052420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.052349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r" event={"ID":"8817e72b-cff4-40ad-96f5-b2ec26b2c5cc","Type":"ContainerDied","Data":"a3ba3f655c64e0c55e21bc03fc0fdc81f969352eefab9f5b93a3d2605c499d7c"}
Sep 15 12:16:18.052420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.052375    2583 scope.go:117] "RemoveContainer" containerID="da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd"
Sep 15 12:16:18.063740 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.063716    2583 scope.go:117] "RemoveContainer" containerID="da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd"
Sep 15 12:16:18.064053 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:16:18.064017    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd\": container with ID starting with da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd not found: ID does not exist" containerID="da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd"
Sep 15 12:16:18.064126 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.064062    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd"} err="failed to get container status \"da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd\": rpc error: code = NotFound desc = could not find container \"da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd\": container with ID starting with da5c9da21f1d0af517bc28ca7d2a9dc3d01a02b5039c78ee716bbb22312f38dd not found: ID does not exist"
Sep 15 12:16:18.072129 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.072064    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:18.075972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.075929    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"]
Sep 15 12:16:18.087577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.087541    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d7eff3e-5c8566f689-4tp8r"]
Sep 15 12:16:18.184496 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.184462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:18.460974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.460937    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:18.716674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.716582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:18.824317 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:18.824283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:19.400494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:19.400445    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:19.807466 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:19.807379    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" path="/var/lib/kubelet/pods/8817e72b-cff4-40ad-96f5-b2ec26b2c5cc/volumes"
Sep 15 12:16:20.310076 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:20.310016    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:21.202444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:21.202410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:21.406474 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:21.406436    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:21.509609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:21.509513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:22.201683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.201629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:22.882821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.882786    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"]
Sep 15 12:16:22.883377 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.883351    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" containerName="maas-api"
Sep 15 12:16:22.883523 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.883504    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" containerName="maas-api"
Sep 15 12:16:22.883751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.883733    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8817e72b-cff4-40ad-96f5-b2ec26b2c5cc" containerName="maas-api"
Sep 15 12:16:22.888882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.888860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:22.891558 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.891535    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1a5ccf40-serving-cert\""
Sep 15 12:16:22.901759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.901732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"]
Sep 15 12:16:22.916864 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.916819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:22.981698 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:22.981661    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:23.009463 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.009433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-metrics-certs\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.009667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.009481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-maas-api-tls\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.009741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.009658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d989226-79ad-45c7-9a01-755b101d5c7e-openshift-service-ca\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.009741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.009728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqx5\" (UniqueName: \"kubernetes.io/projected/4d989226-79ad-45c7-9a01-755b101d5c7e-kube-api-access-mxqx5\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.081493 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.081458    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:23.110652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.110608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d989226-79ad-45c7-9a01-755b101d5c7e-openshift-service-ca\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.110853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.110669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqx5\" (UniqueName: \"kubernetes.io/projected/4d989226-79ad-45c7-9a01-755b101d5c7e-kube-api-access-mxqx5\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.110853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.110733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-metrics-certs\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.110853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.110771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-maas-api-tls\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.111412 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.111385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d989226-79ad-45c7-9a01-755b101d5c7e-openshift-service-ca\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.113505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.113482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-metrics-certs\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.113610 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.113507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-maas-api-tls\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.119326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.119301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqx5\" (UniqueName: \"kubernetes.io/projected/4d989226-79ad-45c7-9a01-755b101d5c7e-kube-api-access-mxqx5\") pod \"maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.202987 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.202899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:23.390140 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.390105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:23.485919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.485479    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"]
Sep 15 12:16:23.492079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.491822    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:23.878575 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:23.878545    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:24.089482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:24.089440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r" event={"ID":"4d989226-79ad-45c7-9a01-755b101d5c7e","Type":"ContainerStarted","Data":"36f22ab7c8d38bc974302359a2159aa11556fac566dda7597e060f5b2b879015"}
Sep 15 12:16:24.089889 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:24.089508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r" event={"ID":"4d989226-79ad-45c7-9a01-755b101d5c7e","Type":"ContainerStarted","Data":"993e66f7fe790fa5935df426ce982f7e7a0b4a1db370a5975eb55e38836ab28a"}
Sep 15 12:16:24.089889 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:24.089641    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:24.110045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:24.109924    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r" podStartSLOduration=1.744957357 podStartE2EDuration="2.109907778s" totalImagesPullingTime="364.950421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:24.107403619 +0000 UTC m=+864.803963214" watchObservedRunningTime="2026-09-15 12:16:24.109907778 +0000 UTC m=+864.806467389"
Sep 15 12:16:24.606249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:24.606215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:25.281811 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:25.281773    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:25.995546 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:25.995511    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:26.683777 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:26.683745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:30.101565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:30.101536    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:39.816646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:39.816606    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:44.966932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:44.966895    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:45.914781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:45.914736    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:46.914239 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:46.914197    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:47.675509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:47.675472    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:48.471525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:48.471486    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:49.274380 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:49.274337    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:49.952628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:49.952592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:50.693336 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:50.693304    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:51.392909 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:51.392875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:52.552714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:52.552624    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.093595 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.093365    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"]
Sep 15 12:16:53.093797 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.093599    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r" podUID="4d989226-79ad-45c7-9a01-755b101d5c7e" containerName="maas-api" containerID="cri-o://36f22ab7c8d38bc974302359a2159aa11556fac566dda7597e060f5b2b879015" gracePeriod=30
Sep 15 12:16:53.122756 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.122721    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.191810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.191776    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.258810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.258781    2583 generic.go:356] "Generic (PLEG): container finished" podID="4d989226-79ad-45c7-9a01-755b101d5c7e" containerID="36f22ab7c8d38bc974302359a2159aa11556fac566dda7597e060f5b2b879015" exitCode=0
Sep 15 12:16:53.258969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.258854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r" event={"ID":"4d989226-79ad-45c7-9a01-755b101d5c7e","Type":"ContainerDied","Data":"36f22ab7c8d38bc974302359a2159aa11556fac566dda7597e060f5b2b879015"}
Sep 15 12:16:53.290958 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.290920    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.345165 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.345093    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:53.375963 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.375917    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.411603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.411547    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-maas-api-tls\") pod \"4d989226-79ad-45c7-9a01-755b101d5c7e\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") "
Sep 15 12:16:53.411780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.411620    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-metrics-certs\") pod \"4d989226-79ad-45c7-9a01-755b101d5c7e\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") "
Sep 15 12:16:53.411830 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.411784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d989226-79ad-45c7-9a01-755b101d5c7e-openshift-service-ca\") pod \"4d989226-79ad-45c7-9a01-755b101d5c7e\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") "
Sep 15 12:16:53.411882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.411845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxqx5\" (UniqueName: \"kubernetes.io/projected/4d989226-79ad-45c7-9a01-755b101d5c7e-kube-api-access-mxqx5\") pod \"4d989226-79ad-45c7-9a01-755b101d5c7e\" (UID: \"4d989226-79ad-45c7-9a01-755b101d5c7e\") "
Sep 15 12:16:53.412343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.412309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d989226-79ad-45c7-9a01-755b101d5c7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d989226-79ad-45c7-9a01-755b101d5c7e" (UID: "4d989226-79ad-45c7-9a01-755b101d5c7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:16:53.414506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.414462    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d989226-79ad-45c7-9a01-755b101d5c7e" (UID: "4d989226-79ad-45c7-9a01-755b101d5c7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:16:53.414618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.414563    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d989226-79ad-45c7-9a01-755b101d5c7e-kube-api-access-mxqx5" (OuterVolumeSpecName: "kube-api-access-mxqx5") pod "4d989226-79ad-45c7-9a01-755b101d5c7e" (UID: "4d989226-79ad-45c7-9a01-755b101d5c7e"). InnerVolumeSpecName "kube-api-access-mxqx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:16:53.414618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.414584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d989226-79ad-45c7-9a01-755b101d5c7e" (UID: "4d989226-79ad-45c7-9a01-755b101d5c7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:16:53.512905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.512863    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d989226-79ad-45c7-9a01-755b101d5c7e-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.512905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.512903    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqx5\" (UniqueName: \"kubernetes.io/projected/4d989226-79ad-45c7-9a01-755b101d5c7e-kube-api-access-mxqx5\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.513196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.512919    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.513196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.512932    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d989226-79ad-45c7-9a01-755b101d5c7e-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:16:53.629188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.629086    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.774579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.774536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:53.964103 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:53.963969    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:54.213485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.213447    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"]
Sep 15 12:16:54.213981 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.213963    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d989226-79ad-45c7-9a01-755b101d5c7e" containerName="maas-api"
Sep 15 12:16:54.214071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.213983    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d989226-79ad-45c7-9a01-755b101d5c7e" containerName="maas-api"
Sep 15 12:16:54.214168 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.214081    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d989226-79ad-45c7-9a01-755b101d5c7e" containerName="maas-api"
Sep 15 12:16:54.217636 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.217597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.220991 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.220965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3998f670-openshift-default-dockercfg-mj9zp\""
Sep 15 12:16:54.228989 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.228969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"]
Sep 15 12:16:54.267332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.267301    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"
Sep 15 12:16:54.267332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.267317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r" event={"ID":"4d989226-79ad-45c7-9a01-755b101d5c7e","Type":"ContainerDied","Data":"993e66f7fe790fa5935df426ce982f7e7a0b4a1db370a5975eb55e38836ab28a"}
Sep 15 12:16:54.267580 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.267373    2583 scope.go:117] "RemoveContainer" containerID="36f22ab7c8d38bc974302359a2159aa11556fac566dda7597e060f5b2b879015"
Sep 15 12:16:54.284754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.284727    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"]
Sep 15 12:16:54.287823 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.287802    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1a5ccf40-5b77d7b876-tt42r"]
Sep 15 12:16:54.321958 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.321889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-data\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.321959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istiod-ca-cert\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-credential-socket\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2np\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-kube-api-access-lh2np\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-envoy\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322383 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-podinfo\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322383 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-token\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322383 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-socket\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.322383 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.322285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-certs\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.422876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.422838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-certs\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423114 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.422962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-data\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423114 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.422990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istiod-ca-cert\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423114 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-credential-socket\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2np\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-kube-api-access-lh2np\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-envoy\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-podinfo\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-token\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-socket\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.423519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-certs\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.424002 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.423970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istiod-ca-cert\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.424154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.424102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-data\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.424154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.424106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-credential-socket\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.424378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.424353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-socket\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.425860 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.425841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-envoy\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.426385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.426363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-podinfo\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.432428 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.432366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-token\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.432882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.432860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2np\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-kube-api-access-lh2np\") pod \"e2e-ait-3998f670-openshift-default-66f9f4d964-4244b\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") " pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.531357 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.531263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:54.817895 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.817860    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"]
Sep 15 12:16:54.819609 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:16:54.819583    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ab2068_e236_4a97_86da_a3a5c4f51c8a.slice/crio-c6f3c97a9723908f51dc479684f51f8ef44033bbe63deb1b9d545d969050b4ca WatchSource:0}: Error finding container c6f3c97a9723908f51dc479684f51f8ef44033bbe63deb1b9d545d969050b4ca: Status 404 returned error can't find the container with id c6f3c97a9723908f51dc479684f51f8ef44033bbe63deb1b9d545d969050b4ca
Sep 15 12:16:54.822754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.822721    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:54.822845 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.822787    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:54.822845 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.822824    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:16:54.893608 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:54.893567    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:55.036942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.036910    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:55.276528 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.276438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" event={"ID":"66ab2068-e236-4a97-86da-a3a5c4f51c8a","Type":"ContainerStarted","Data":"af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"}
Sep 15 12:16:55.276528 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.276495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" event={"ID":"66ab2068-e236-4a97-86da-a3a5c4f51c8a","Type":"ContainerStarted","Data":"c6f3c97a9723908f51dc479684f51f8ef44033bbe63deb1b9d545d969050b4ca"}
Sep 15 12:16:55.296958 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.296907    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podStartSLOduration=1.296891151 podStartE2EDuration="1.296891151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:16:55.29657998 +0000 UTC m=+895.993139580" watchObservedRunningTime="2026-09-15 12:16:55.296891151 +0000 UTC m=+895.993450728"
Sep 15 12:16:55.333955 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.333918    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:55.531952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.531870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:55.537256 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.537230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:55.807602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:55.807514    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d989226-79ad-45c7-9a01-755b101d5c7e" path="/var/lib/kubelet/pods/4d989226-79ad-45c7-9a01-755b101d5c7e/volumes"
Sep 15 12:16:56.212913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:56.212876    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:56.281720 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:56.281693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:56.282543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:56.282524    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:16:56.648470 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:56.648437    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:56.734064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:56.734014    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:57.076215 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:57.076179    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:57.870060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:57.870005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:58.569520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.569479    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"]
Sep 15 12:16:58.573528 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.573504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.575954 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.575929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3998f670-serving-cert\""
Sep 15 12:16:58.587462 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.587431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"]
Sep 15 12:16:58.599005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.598971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:58.665278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.665241    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-metrics-certs\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.665278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.665280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47c45\" (UniqueName: \"kubernetes.io/projected/f6b11baa-8477-496d-8d11-77ad2ea53042-kube-api-access-47c45\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.665497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.665352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.665497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.665429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b11baa-8477-496d-8d11-77ad2ea53042-openshift-service-ca\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.700702 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.700664    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:58.766331 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.766298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.766520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.766380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b11baa-8477-496d-8d11-77ad2ea53042-openshift-service-ca\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.766520 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:16:58.766467    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-serving-cert: secret "maas-api-e2e-ait-3998f670-serving-cert" not found
Sep 15 12:16:58.766520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.766484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-metrics-certs\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.766520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.766517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47c45\" (UniqueName: \"kubernetes.io/projected/f6b11baa-8477-496d-8d11-77ad2ea53042-kube-api-access-47c45\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.766719 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:16:58.766549    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls podName:f6b11baa-8477-496d-8d11-77ad2ea53042 nodeName:}" failed. No retries permitted until 2026-09-15 12:16:59.266526961 +0000 UTC m=+899.963086534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls") pod "maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" (UID: "f6b11baa-8477-496d-8d11-77ad2ea53042") : secret "maas-api-e2e-ait-3998f670-serving-cert" not found
Sep 15 12:16:58.767131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.767107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b11baa-8477-496d-8d11-77ad2ea53042-openshift-service-ca\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.769041 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.769007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-metrics-certs\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:58.776787 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:58.776760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47c45\" (UniqueName: \"kubernetes.io/projected/f6b11baa-8477-496d-8d11-77ad2ea53042-kube-api-access-47c45\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:59.021776 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.021687    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:59.129999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.129959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:59.206994 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.206957    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:59.271394 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.271357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:59.273803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.273742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls\") pod \"maas-api-e2e-ait-3998f670-5974cc469d-4c5p8\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:59.486498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.486464    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:16:59.502295 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.502251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:16:59.763017 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.762990    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"]
Sep 15 12:16:59.764601 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:16:59.764563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b11baa_8477_496d_8d11_77ad2ea53042.slice/crio-6cf80d88a69062a0e56836afa14bd5f05fb186c2efc401119f180d1649dc0fe9 WatchSource:0}: Error finding container 6cf80d88a69062a0e56836afa14bd5f05fb186c2efc401119f180d1649dc0fe9: Status 404 returned error can't find the container with id 6cf80d88a69062a0e56836afa14bd5f05fb186c2efc401119f180d1649dc0fe9
Sep 15 12:16:59.777310 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.777285    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:16:59.777676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:16:59.777658    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:17:00.215667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:00.215577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:00.307289 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:00.307249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" event={"ID":"f6b11baa-8477-496d-8d11-77ad2ea53042","Type":"ContainerStarted","Data":"6cf80d88a69062a0e56836afa14bd5f05fb186c2efc401119f180d1649dc0fe9"}
Sep 15 12:17:00.975631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:00.975594    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:01.316461 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:01.316432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" event={"ID":"f6b11baa-8477-496d-8d11-77ad2ea53042","Type":"ContainerStarted","Data":"9208a098cfeab2730ef438fcad80f10b17234279cce282540e5daf2a97925c36"}
Sep 15 12:17:01.316864 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:01.316489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:17:01.337989 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:01.337942    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" podStartSLOduration=2.827180369 podStartE2EDuration="3.337928628s" totalImagesPullingTime="510.748259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:01.33641895 +0000 UTC m=+902.032978548" watchObservedRunningTime="2026-09-15 12:17:01.337928628 +0000 UTC m=+902.034488206"
Sep 15 12:17:01.671276 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:01.671149    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:07.331377 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:07.331342    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:17:07.819108 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:07.805287    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:15.394224 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:15.394187    2583 generic.go:356] "Generic (PLEG): container finished" podID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerID="af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350" exitCode=0
Sep 15 12:17:15.394647 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:15.394259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" event={"ID":"66ab2068-e236-4a97-86da-a3a5c4f51c8a","Type":"ContainerDied","Data":"af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"}
Sep 15 12:17:15.394697 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:15.394682    2583 scope.go:117] "RemoveContainer" containerID="af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"
Sep 15 12:17:15.395828 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:15.395802    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:15.395881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:15.395867    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:15.395917 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:15.395897    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:17:16.401672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:16.401634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" event={"ID":"66ab2068-e236-4a97-86da-a3a5c4f51c8a","Type":"ContainerStarted","Data":"73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf"}
Sep 15 12:17:16.532158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:16.532112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:17:16.532431 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:16.532407    2583 patch_prober.go:28] interesting pod/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 12:17:16.532489 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:16.532459    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 12:17:17.532497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:17.532443    2583 patch_prober.go:28] interesting pod/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 12:17:17.532995 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:17.532550    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 12:17:18.532498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:18.532460    2583 patch_prober.go:28] interesting pod/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 12:17:18.532714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:18.532557    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 12:17:19.532299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:19.532255    2583 patch_prober.go:28] interesting pod/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 12:17:19.532565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:19.532326    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 12:17:20.532098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:20.532052    2583 patch_prober.go:28] interesting pod/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 12:17:20.532631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:20.532147    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 12:17:20.691181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:20.691139    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:21.510764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:21.510719    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:21.532726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:21.532690    2583 patch_prober.go:28] interesting pod/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 15 12:17:21.533119 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:21.532765    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 15 12:17:22.241562 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:22.241523    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:22.535973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:22.535888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:17:22.536463 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:22.536321    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:17:22.536893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:22.536874    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:17:23.198766 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:23.198733    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:23.865919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:23.865883    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:24.628533 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:24.628501    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:25.334744 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:25.334709    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:25.641878 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:25.641781    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:26.114982 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:26.114944    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:26.803318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:26.803283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:26.936142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:26.936105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:27.672581 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:27.672545    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:28.362851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.362815    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"]
Sep 15 12:17:28.363380 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.363100    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" podUID="f6b11baa-8477-496d-8d11-77ad2ea53042" containerName="maas-api" containerID="cri-o://9208a098cfeab2730ef438fcad80f10b17234279cce282540e5daf2a97925c36" gracePeriod=30
Sep 15 12:17:28.391586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.391548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:28.467318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.467279    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:28.482715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.482691    2583 generic.go:356] "Generic (PLEG): container finished" podID="f6b11baa-8477-496d-8d11-77ad2ea53042" containerID="9208a098cfeab2730ef438fcad80f10b17234279cce282540e5daf2a97925c36" exitCode=0
Sep 15 12:17:28.482815 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.482768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" event={"ID":"f6b11baa-8477-496d-8d11-77ad2ea53042","Type":"ContainerDied","Data":"9208a098cfeab2730ef438fcad80f10b17234279cce282540e5daf2a97925c36"}
Sep 15 12:17:28.549548 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.549355    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:28.621809 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.621736    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:17:28.665700 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.665670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-metrics-certs\") pod \"f6b11baa-8477-496d-8d11-77ad2ea53042\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") "
Sep 15 12:17:28.665870 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.665708    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47c45\" (UniqueName: \"kubernetes.io/projected/f6b11baa-8477-496d-8d11-77ad2ea53042-kube-api-access-47c45\") pod \"f6b11baa-8477-496d-8d11-77ad2ea53042\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") "
Sep 15 12:17:28.665870 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.665764    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls\") pod \"f6b11baa-8477-496d-8d11-77ad2ea53042\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") "
Sep 15 12:17:28.666810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.666717    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b11baa-8477-496d-8d11-77ad2ea53042-openshift-service-ca\") pod \"f6b11baa-8477-496d-8d11-77ad2ea53042\" (UID: \"f6b11baa-8477-496d-8d11-77ad2ea53042\") "
Sep 15 12:17:28.667226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.667190    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b11baa-8477-496d-8d11-77ad2ea53042-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6b11baa-8477-496d-8d11-77ad2ea53042" (UID: "f6b11baa-8477-496d-8d11-77ad2ea53042"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:28.669182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.669151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6b11baa-8477-496d-8d11-77ad2ea53042" (UID: "f6b11baa-8477-496d-8d11-77ad2ea53042"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:17:28.669182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.669165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b11baa-8477-496d-8d11-77ad2ea53042-kube-api-access-47c45" (OuterVolumeSpecName: "kube-api-access-47c45") pod "f6b11baa-8477-496d-8d11-77ad2ea53042" (UID: "f6b11baa-8477-496d-8d11-77ad2ea53042"). InnerVolumeSpecName "kube-api-access-47c45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:28.669306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.669175    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6b11baa-8477-496d-8d11-77ad2ea53042" (UID: "f6b11baa-8477-496d-8d11-77ad2ea53042"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:17:28.768156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.768118    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.768156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.768150    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47c45\" (UniqueName: \"kubernetes.io/projected/f6b11baa-8477-496d-8d11-77ad2ea53042-kube-api-access-47c45\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.768156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.768160    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b11baa-8477-496d-8d11-77ad2ea53042-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.768389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.768169    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b11baa-8477-496d-8d11-77ad2ea53042-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:28.878081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.877960    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:28.973467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:28.973417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:29.078749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.078716    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:29.490072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.490043    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"
Sep 15 12:17:29.490072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.490053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8" event={"ID":"f6b11baa-8477-496d-8d11-77ad2ea53042","Type":"ContainerDied","Data":"6cf80d88a69062a0e56836afa14bd5f05fb186c2efc401119f180d1649dc0fe9"}
Sep 15 12:17:29.490561 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.490118    2583 scope.go:117] "RemoveContainer" containerID="9208a098cfeab2730ef438fcad80f10b17234279cce282540e5daf2a97925c36"
Sep 15 12:17:29.512572 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.512543    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"]
Sep 15 12:17:29.519143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.519115    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3998f670-5974cc469d-4c5p8"]
Sep 15 12:17:29.759080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.758967    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"]
Sep 15 12:17:29.759282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.759229    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy" containerID="cri-o://73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf" gracePeriod=30
Sep 15 12:17:29.772434 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.772407    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:29.807868 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.807838    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b11baa-8477-496d-8d11-77ad2ea53042" path="/var/lib/kubelet/pods/f6b11baa-8477-496d-8d11-77ad2ea53042/volumes"
Sep 15 12:17:29.873050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:29.872947    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:30.241686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:30.241639    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:30.548245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:30.548145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:30.716190 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:30.716152    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:30.849582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:30.849551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:31.429071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:31.429005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:33.982693 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:33.982659    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"]
Sep 15 12:17:33.983362 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:33.983340    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b11baa-8477-496d-8d11-77ad2ea53042" containerName="maas-api"
Sep 15 12:17:33.983460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:33.983363    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b11baa-8477-496d-8d11-77ad2ea53042" containerName="maas-api"
Sep 15 12:17:33.983524 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:33.983473    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b11baa-8477-496d-8d11-77ad2ea53042" containerName="maas-api"
Sep 15 12:17:33.988095 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:33.988073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:33.992082 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:33.992006    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-66020bb6-serving-cert\""
Sep 15 12:17:34.008712 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.008682    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"]
Sep 15 12:17:34.023451 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.023419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f74s\" (UniqueName: \"kubernetes.io/projected/ea7db4fe-832c-4069-b55a-588cff26d4d1-kube-api-access-8f74s\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.023616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.023486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.023616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.023563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7db4fe-832c-4069-b55a-588cff26d4d1-openshift-service-ca\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.023705 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.023627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-metrics-certs\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.124315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.124277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f74s\" (UniqueName: \"kubernetes.io/projected/ea7db4fe-832c-4069-b55a-588cff26d4d1-kube-api-access-8f74s\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.124535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.124336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.124535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.124383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7db4fe-832c-4069-b55a-588cff26d4d1-openshift-service-ca\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.124535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.124431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-metrics-certs\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.124535 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:17:34.124504    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-serving-cert: secret "maas-api-e2e-derive-66020bb6-serving-cert" not found
Sep 15 12:17:34.124742 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:17:34.124592    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls podName:ea7db4fe-832c-4069-b55a-588cff26d4d1 nodeName:}" failed. No retries permitted until 2026-09-15 12:17:34.62457309 +0000 UTC m=+935.321132648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls") pod "maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" (UID: "ea7db4fe-832c-4069-b55a-588cff26d4d1") : secret "maas-api-e2e-derive-66020bb6-serving-cert" not found
Sep 15 12:17:34.125049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.125016    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7db4fe-832c-4069-b55a-588cff26d4d1-openshift-service-ca\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.127075 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.127053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-metrics-certs\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.145295 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.145265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f74s\" (UniqueName: \"kubernetes.io/projected/ea7db4fe-832c-4069-b55a-588cff26d4d1-kube-api-access-8f74s\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.299643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.299554    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:34.393544 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.393500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:34.630092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.630017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.632504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.632474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls\") pod \"maas-api-e2e-derive-66020bb6-77b7568667-pb7tb\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:34.874689 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.874663    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:34.900639 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:34.900437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:35.039070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.039018    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:17:35.135453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istiod-ca-cert\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135464    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-podinfo\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-socket\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135588    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-credential-socket\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135618    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-certs\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-data\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135668    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-token\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-envoy\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.135774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.135753    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh2np\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-kube-api-access-lh2np\") pod \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\" (UID: \"66ab2068-e236-4a97-86da-a3a5c4f51c8a\") "
Sep 15 12:17:35.136487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:17:35.136487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:17:35.136487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136383    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.136487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136410    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.136487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136384    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:17:35.136487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136440    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:17:35.136975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.136861    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:35.138697 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.138673    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:17:35.138972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.138740    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:35.139134 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.139110    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:17:35.139260 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.139241    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-kube-api-access-lh2np" (OuterVolumeSpecName: "kube-api-access-lh2np") pod "66ab2068-e236-4a97-86da-a3a5c4f51c8a" (UID: "66ab2068-e236-4a97-86da-a3a5c4f51c8a"). InnerVolumeSpecName "kube-api-access-lh2np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:35.200987 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.200949    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"]
Sep 15 12:17:35.201525 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:17:35.201492    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7db4fe_832c_4069_b55a_588cff26d4d1.slice/crio-fad31a672f0ac155491405cd0a9286c1eb99b0599cbf68208df631b0dcad91dd WatchSource:0}: Error finding container fad31a672f0ac155491405cd0a9286c1eb99b0599cbf68208df631b0dcad91dd: Status 404 returned error can't find the container with id fad31a672f0ac155491405cd0a9286c1eb99b0599cbf68208df631b0dcad91dd
Sep 15 12:17:35.237680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237641    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.237680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237677    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.237680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237686    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.237906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237695    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.237906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237705    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.237906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237712    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ab2068-e236-4a97-86da-a3a5c4f51c8a-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.237906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.237720    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2np\" (UniqueName: \"kubernetes.io/projected/66ab2068-e236-4a97-86da-a3a5c4f51c8a-kube-api-access-lh2np\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:35.529110 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.528995    2583 generic.go:356] "Generic (PLEG): container finished" podID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerID="73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf" exitCode=0
Sep 15 12:17:35.529110 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.529065    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"
Sep 15 12:17:35.529340 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.529063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" event={"ID":"66ab2068-e236-4a97-86da-a3a5c4f51c8a","Type":"ContainerDied","Data":"73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf"}
Sep 15 12:17:35.529340 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.529211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b" event={"ID":"66ab2068-e236-4a97-86da-a3a5c4f51c8a","Type":"ContainerDied","Data":"c6f3c97a9723908f51dc479684f51f8ef44033bbe63deb1b9d545d969050b4ca"}
Sep 15 12:17:35.529340 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.529248    2583 scope.go:117] "RemoveContainer" containerID="73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf"
Sep 15 12:17:35.530536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.530513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" event={"ID":"ea7db4fe-832c-4069-b55a-588cff26d4d1","Type":"ContainerStarted","Data":"fad31a672f0ac155491405cd0a9286c1eb99b0599cbf68208df631b0dcad91dd"}
Sep 15 12:17:35.539692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.539661    2583 scope.go:117] "RemoveContainer" containerID="af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"
Sep 15 12:17:35.548732 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.548714    2583 scope.go:117] "RemoveContainer" containerID="73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf"
Sep 15 12:17:35.549017 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:17:35.548997    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf\": container with ID starting with 73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf not found: ID does not exist" containerID="73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf"
Sep 15 12:17:35.549101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.549042    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf"} err="failed to get container status \"73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf\": rpc error: code = NotFound desc = could not find container \"73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf\": container with ID starting with 73a109896126d3f33d7256c869c3a9ffae0065a4a36dc8783d4e3f8cc2643aaf not found: ID does not exist"
Sep 15 12:17:35.549101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.549062    2583 scope.go:117] "RemoveContainer" containerID="af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"
Sep 15 12:17:35.549328 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:17:35.549310    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350\": container with ID starting with af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350 not found: ID does not exist" containerID="af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"
Sep 15 12:17:35.549380 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.549334    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350"} err="failed to get container status \"af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350\": rpc error: code = NotFound desc = could not find container \"af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350\": container with ID starting with af714852c055ffa0ac43cad9155dfe6927303f757f6fae64c2458c4c25507350 not found: ID does not exist"
Sep 15 12:17:35.559514 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.559484    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"]
Sep 15 12:17:35.566517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.566494    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3998f670-openshift-default-66f9f4d964-4244b"]
Sep 15 12:17:35.807810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:35.807778    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" path="/var/lib/kubelet/pods/66ab2068-e236-4a97-86da-a3a5c4f51c8a/volumes"
Sep 15 12:17:36.305653 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:36.305614    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:36.405600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:36.405567    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:36.507439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:36.507396    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:36.538948 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:36.538910    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" event={"ID":"ea7db4fe-832c-4069-b55a-588cff26d4d1","Type":"ContainerStarted","Data":"aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800"}
Sep 15 12:17:36.539164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:36.539006    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:36.559393 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:36.559270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" podStartSLOduration=3.022160121 podStartE2EDuration="3.559249535s" totalImagesPullingTime="537.089414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:17:36.558657142 +0000 UTC m=+937.255216720" watchObservedRunningTime="2026-09-15 12:17:36.559249535 +0000 UTC m=+937.255809114"
Sep 15 12:17:37.132369 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:37.132322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:37.236902 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:37.236866    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:37.309879 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:37.309833    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:38.169293 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:38.169254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:38.846220 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:38.846185    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:42.550832 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:42.550799    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:44.066792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:44.066747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:44.723284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:44.723236    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:45.481045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:45.480987    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:46.303467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:46.303435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:47.019851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:47.019818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:47.889676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:47.889639    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:48.821479 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:48.821447    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:49.539283 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:49.539246    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:50.305811 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:50.305778    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:51.009337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:51.009300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:51.811427 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:51.811391    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:52.545232 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:52.545193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:53.297178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:53.297135    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:54.102632 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.102603    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"]
Sep 15 12:17:54.102857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.102832    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr" podUID="65a614b8-6ac0-4a39-9642-a28dad3159d5" containerName="maas-api" containerID="cri-o://e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5" gracePeriod=30
Sep 15 12:17:54.363072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.362996    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:17:54.529980 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.529947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-maas-api-tls\") pod \"65a614b8-6ac0-4a39-9642-a28dad3159d5\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") "
Sep 15 12:17:54.530202 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.529993    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-metrics-certs\") pod \"65a614b8-6ac0-4a39-9642-a28dad3159d5\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") "
Sep 15 12:17:54.530202 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.530110    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a614b8-6ac0-4a39-9642-a28dad3159d5-openshift-service-ca\") pod \"65a614b8-6ac0-4a39-9642-a28dad3159d5\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") "
Sep 15 12:17:54.530202 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.530169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx88x\" (UniqueName: \"kubernetes.io/projected/65a614b8-6ac0-4a39-9642-a28dad3159d5-kube-api-access-zx88x\") pod \"65a614b8-6ac0-4a39-9642-a28dad3159d5\" (UID: \"65a614b8-6ac0-4a39-9642-a28dad3159d5\") "
Sep 15 12:17:54.530703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.530654    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a614b8-6ac0-4a39-9642-a28dad3159d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65a614b8-6ac0-4a39-9642-a28dad3159d5" (UID: "65a614b8-6ac0-4a39-9642-a28dad3159d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:54.532824 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.532788    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65a614b8-6ac0-4a39-9642-a28dad3159d5" (UID: "65a614b8-6ac0-4a39-9642-a28dad3159d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:17:54.533012 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.532985    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a614b8-6ac0-4a39-9642-a28dad3159d5-kube-api-access-zx88x" (OuterVolumeSpecName: "kube-api-access-zx88x") pod "65a614b8-6ac0-4a39-9642-a28dad3159d5" (UID: "65a614b8-6ac0-4a39-9642-a28dad3159d5"). InnerVolumeSpecName "kube-api-access-zx88x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:54.533228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.533207    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65a614b8-6ac0-4a39-9642-a28dad3159d5" (UID: "65a614b8-6ac0-4a39-9642-a28dad3159d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:17:54.631643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.631564    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:54.631643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.631591    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65a614b8-6ac0-4a39-9642-a28dad3159d5-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:54.631643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.631601    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a614b8-6ac0-4a39-9642-a28dad3159d5-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:54.631643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.631610    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx88x\" (UniqueName: \"kubernetes.io/projected/65a614b8-6ac0-4a39-9642-a28dad3159d5-kube-api-access-zx88x\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:54.647939 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.647901    2583 generic.go:356] "Generic (PLEG): container finished" podID="65a614b8-6ac0-4a39-9642-a28dad3159d5" containerID="e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5" exitCode=0
Sep 15 12:17:54.648143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.647954    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"
Sep 15 12:17:54.648143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.647993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr" event={"ID":"65a614b8-6ac0-4a39-9642-a28dad3159d5","Type":"ContainerDied","Data":"e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5"}
Sep 15 12:17:54.648143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.648080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr" event={"ID":"65a614b8-6ac0-4a39-9642-a28dad3159d5","Type":"ContainerDied","Data":"982a8692096238211d00f17688808bd3fccd8aade727b7beb315d1aff15c42bc"}
Sep 15 12:17:54.648143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.648106    2583 scope.go:117] "RemoveContainer" containerID="e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5"
Sep 15 12:17:54.658243 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.658227    2583 scope.go:117] "RemoveContainer" containerID="e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5"
Sep 15 12:17:54.658511 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:17:54.658490    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5\": container with ID starting with e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5 not found: ID does not exist" containerID="e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5"
Sep 15 12:17:54.658558 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.658520    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5"} err="failed to get container status \"e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5\": rpc error: code = NotFound desc = could not find container \"e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5\": container with ID starting with e10529fa97fb97dffd2fa8efececd8a48bfceecc8782327bdb0c1fafea493dc5 not found: ID does not exist"
Sep 15 12:17:54.678001 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.677961    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"]
Sep 15 12:17:54.679978 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:54.679959    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0e10f0-965c475df-drxgr"]
Sep 15 12:17:55.257562 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:55.257524    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:55.318553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:55.318510    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:55.606348 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:55.606313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:55.807303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:55.807267    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a614b8-6ac0-4a39-9642-a28dad3159d5" path="/var/lib/kubelet/pods/65a614b8-6ac0-4a39-9642-a28dad3159d5/volumes"
Sep 15 12:17:56.364157 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:56.364118    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:57.342505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:57.342470    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:58.078703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:58.078670    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:17:58.831547 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:58.831517    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"]
Sep 15 12:17:58.831966 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:58.831733    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" podUID="ea7db4fe-832c-4069-b55a-588cff26d4d1" containerName="maas-api" containerID="cri-o://aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800" gracePeriod=30
Sep 15 12:17:59.096950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.096890    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:59.176605 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.176572    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-metrics-certs\") pod \"ea7db4fe-832c-4069-b55a-588cff26d4d1\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") "
Sep 15 12:17:59.176802 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.176633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f74s\" (UniqueName: \"kubernetes.io/projected/ea7db4fe-832c-4069-b55a-588cff26d4d1-kube-api-access-8f74s\") pod \"ea7db4fe-832c-4069-b55a-588cff26d4d1\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") "
Sep 15 12:17:59.176802 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.176664    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7db4fe-832c-4069-b55a-588cff26d4d1-openshift-service-ca\") pod \"ea7db4fe-832c-4069-b55a-588cff26d4d1\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") "
Sep 15 12:17:59.176802 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.176725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls\") pod \"ea7db4fe-832c-4069-b55a-588cff26d4d1\" (UID: \"ea7db4fe-832c-4069-b55a-588cff26d4d1\") "
Sep 15 12:17:59.177088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.177045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7db4fe-832c-4069-b55a-588cff26d4d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea7db4fe-832c-4069-b55a-588cff26d4d1" (UID: "ea7db4fe-832c-4069-b55a-588cff26d4d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:17:59.178837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.178808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea7db4fe-832c-4069-b55a-588cff26d4d1" (UID: "ea7db4fe-832c-4069-b55a-588cff26d4d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:17:59.178837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.178823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7db4fe-832c-4069-b55a-588cff26d4d1-kube-api-access-8f74s" (OuterVolumeSpecName: "kube-api-access-8f74s") pod "ea7db4fe-832c-4069-b55a-588cff26d4d1" (UID: "ea7db4fe-832c-4069-b55a-588cff26d4d1"). InnerVolumeSpecName "kube-api-access-8f74s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:17:59.178967 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.178929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea7db4fe-832c-4069-b55a-588cff26d4d1" (UID: "ea7db4fe-832c-4069-b55a-588cff26d4d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:17:59.277536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.277505    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:59.277536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.277533    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7db4fe-832c-4069-b55a-588cff26d4d1-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:59.277536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.277543    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f74s\" (UniqueName: \"kubernetes.io/projected/ea7db4fe-832c-4069-b55a-588cff26d4d1-kube-api-access-8f74s\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:59.277766 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.277553    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7db4fe-832c-4069-b55a-588cff26d4d1-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:17:59.678216 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.678186    2583 generic.go:356] "Generic (PLEG): container finished" podID="ea7db4fe-832c-4069-b55a-588cff26d4d1" containerID="aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800" exitCode=0
Sep 15 12:17:59.678216 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.678228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" event={"ID":"ea7db4fe-832c-4069-b55a-588cff26d4d1","Type":"ContainerDied","Data":"aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800"}
Sep 15 12:17:59.678472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.678232    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"
Sep 15 12:17:59.678472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.678265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb" event={"ID":"ea7db4fe-832c-4069-b55a-588cff26d4d1","Type":"ContainerDied","Data":"fad31a672f0ac155491405cd0a9286c1eb99b0599cbf68208df631b0dcad91dd"}
Sep 15 12:17:59.678472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.678288    2583 scope.go:117] "RemoveContainer" containerID="aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800"
Sep 15 12:17:59.687833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.687810    2583 scope.go:117] "RemoveContainer" containerID="aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800"
Sep 15 12:17:59.688124 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:17:59.688105    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800\": container with ID starting with aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800 not found: ID does not exist" containerID="aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800"
Sep 15 12:17:59.688186 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.688132    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800"} err="failed to get container status \"aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800\": rpc error: code = NotFound desc = could not find container \"aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800\": container with ID starting with aeee297828164d5d4a53aed0ab0ecdd057bccbb72c1c7b7092bd7442d6ff1800 not found: ID does not exist"
Sep 15 12:17:59.701260 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.701230    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"]
Sep 15 12:17:59.705494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.705474    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-66020bb6-77b7568667-pb7tb"]
Sep 15 12:17:59.815950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.815911    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7db4fe-832c-4069-b55a-588cff26d4d1" path="/var/lib/kubelet/pods/ea7db4fe-832c-4069-b55a-588cff26d4d1/volumes"
Sep 15 12:17:59.952532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:17:59.952432    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:00.068040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:00.067995    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:00.167060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:00.166998    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:00.603706 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:00.603672    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:01.344564 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:01.344530    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:01.681323 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:01.681237    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:02.189557 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:02.189515    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:03.103743 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:03.103705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:07.852874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:07.852833    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:07.999471 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:07.999431    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:08.229509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:08.229414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:08.467686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:08.467656    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:08.651176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:08.651140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:28.993942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:28.993904    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:33.923593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:33.923556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:34.700194 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:34.700157    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:40.574929 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:40.574884    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:40.742829 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:40.742785    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:45.091189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:45.091154    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:47.423284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:47.423250    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:47.720732 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:47.720642    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:48.153522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.153488    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:48.565894 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.565869    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"]
Sep 15 12:18:48.566326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566312    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy"
Sep 15 12:18:48.566370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy"
Sep 15 12:18:48.566370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566338    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7db4fe-832c-4069-b55a-588cff26d4d1" containerName="maas-api"
Sep 15 12:18:48.566370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566344    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7db4fe-832c-4069-b55a-588cff26d4d1" containerName="maas-api"
Sep 15 12:18:48.566370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566367    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a614b8-6ac0-4a39-9642-a28dad3159d5" containerName="maas-api"
Sep 15 12:18:48.566497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566374    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a614b8-6ac0-4a39-9642-a28dad3159d5" containerName="maas-api"
Sep 15 12:18:48.566497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566434    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7db4fe-832c-4069-b55a-588cff26d4d1" containerName="maas-api"
Sep 15 12:18:48.566497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566443    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a614b8-6ac0-4a39-9642-a28dad3159d5" containerName="maas-api"
Sep 15 12:18:48.566497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.566448    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy"
Sep 15 12:18:48.569956 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.569937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.572532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.572510    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-294781-openshift-default-dockercfg-p2lln\""
Sep 15 12:18:48.608731 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.608690    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"]
Sep 15 12:18:48.618388 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.618354    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:48.633468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-credential-socket\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3663b0cd-e965-48d1-be03-2184715d86d3-istiod-ca-cert\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3663b0cd-e965-48d1-be03-2184715d86d3-istio-podinfo\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-istio-token\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-envoy\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-data\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-socket\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj87h\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-kube-api-access-lj87h\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.633999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.633916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-certs\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.734625 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-data\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.734803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-socket\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.734803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj87h\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-kube-api-access-lj87h\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.734803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-certs\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.734965 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-credential-socket\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.734965 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3663b0cd-e965-48d1-be03-2184715d86d3-istiod-ca-cert\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3663b0cd-e965-48d1-be03-2184715d86d3-istio-podinfo\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-istio-token\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.734999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-data\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.735058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-socket\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.735103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-envoy\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735351 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.735105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-certs\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735351 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.735182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-credential-socket\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.735718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.735697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3663b0cd-e965-48d1-be03-2184715d86d3-istiod-ca-cert\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.737346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.737317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-envoy\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.737579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.737562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3663b0cd-e965-48d1-be03-2184715d86d3-istio-podinfo\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.746558 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.746537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-istio-token\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.747162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.747146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj87h\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-kube-api-access-lj87h\") pod \"e2e-explicit-294781-openshift-default-8774d4c84-wkjrw\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") " pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:48.829653 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.829540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:48.880800 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:48.880760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:49.168916 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.168887    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"]
Sep 15 12:18:49.170508 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:18:49.170482    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3663b0cd_e965_48d1_be03_2184715d86d3.slice/crio-cef02ab2c7110852508b8ded957854e0b9d7934d368254690d5e41fde0f6e88a WatchSource:0}: Error finding container cef02ab2c7110852508b8ded957854e0b9d7934d368254690d5e41fde0f6e88a: Status 404 returned error can't find the container with id cef02ab2c7110852508b8ded957854e0b9d7934d368254690d5e41fde0f6e88a
Sep 15 12:18:49.172786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.172756    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:18:49.172856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.172833    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:18:49.172897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.172864    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:18:49.948306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.948272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" event={"ID":"3663b0cd-e965-48d1-be03-2184715d86d3","Type":"ContainerStarted","Data":"9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"}
Sep 15 12:18:49.948687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.948325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" event={"ID":"3663b0cd-e965-48d1-be03-2184715d86d3","Type":"ContainerStarted","Data":"cef02ab2c7110852508b8ded957854e0b9d7934d368254690d5e41fde0f6e88a"}
Sep 15 12:18:49.984484 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:49.979490    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podStartSLOduration=1.9794694160000001 podStartE2EDuration="1.979469416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:49.974907428 +0000 UTC m=+1010.671467026" watchObservedRunningTime="2026-09-15 12:18:49.979469416 +0000 UTC m=+1010.676028996"
Sep 15 12:18:50.881092 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:50.881050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:50.886250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:50.886224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:50.955066 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:50.955003    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:50.956158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:50.956127    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:18:51.377599 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:51.377563    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:52.196709 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:52.196676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:52.335306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:52.335264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:52.516164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:52.516070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:53.735681 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:53.735643    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:54.788326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:54.788291    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:55.830530 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:55.830489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:56.548293 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.548255    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:56.748776 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.748743    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"]
Sep 15 12:18:56.749495 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.749474    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy"
Sep 15 12:18:56.749553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.749501    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy"
Sep 15 12:18:56.749652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.749630    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ab2068-e236-4a97-86da-a3a5c4f51c8a" containerName="istio-proxy"
Sep 15 12:18:56.753133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.753111    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.756943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.756917    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-294781-serving-cert\""
Sep 15 12:18:56.766183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.766157    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"]
Sep 15 12:18:56.803851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.803765    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:56.812072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.812016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-metrics-certs\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.812237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.812108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00eed17-42bc-4585-90d5-2730249314dc-openshift-service-ca\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.812237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.812151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.812237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.812223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzx2\" (UniqueName: \"kubernetes.io/projected/e00eed17-42bc-4585-90d5-2730249314dc-kube-api-access-ppzx2\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.912840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.912803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:56.913349 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.912926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-metrics-certs\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.913349 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.912969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00eed17-42bc-4585-90d5-2730249314dc-openshift-service-ca\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.913349 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.913009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.913349 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:18:56.913117    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-294781-serving-cert: secret "maas-api-e2e-explicit-294781-serving-cert" not found
Sep 15 12:18:56.913349 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:18:56.913190    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls podName:e00eed17-42bc-4585-90d5-2730249314dc nodeName:}" failed. No retries permitted until 2026-09-15 12:18:57.413163848 +0000 UTC m=+1018.109723403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls") pod "maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" (UID: "e00eed17-42bc-4585-90d5-2730249314dc") : secret "maas-api-e2e-explicit-294781-serving-cert" not found
Sep 15 12:18:56.913349 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.913250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzx2\" (UniqueName: \"kubernetes.io/projected/e00eed17-42bc-4585-90d5-2730249314dc-kube-api-access-ppzx2\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.913843 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.913822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00eed17-42bc-4585-90d5-2730249314dc-openshift-service-ca\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.915550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.915528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-metrics-certs\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:56.929728 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:56.929703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzx2\" (UniqueName: \"kubernetes.io/projected/e00eed17-42bc-4585-90d5-2730249314dc-kube-api-access-ppzx2\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:57.388987 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.388938    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:57.418546 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.418504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:57.420912 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.420887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls\") pod \"maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:57.493494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.493449    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:18:57.667238 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.667153    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:57.950412 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.950385    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"]
Sep 15 12:18:57.951843 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:18:57.951812    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00eed17_42bc_4585_90d5_2730249314dc.slice/crio-c5416078ebc43ae483f4f49a5497e3d8775880f35647d6df61e892d9496c691b WatchSource:0}: Error finding container c5416078ebc43ae483f4f49a5497e3d8775880f35647d6df61e892d9496c691b: Status 404 returned error can't find the container with id c5416078ebc43ae483f4f49a5497e3d8775880f35647d6df61e892d9496c691b
Sep 15 12:18:57.996109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:57.996072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" event={"ID":"e00eed17-42bc-4585-90d5-2730249314dc","Type":"ContainerStarted","Data":"c5416078ebc43ae483f4f49a5497e3d8775880f35647d6df61e892d9496c691b"}
Sep 15 12:18:59.003554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:59.003514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" event={"ID":"e00eed17-42bc-4585-90d5-2730249314dc","Type":"ContainerStarted","Data":"99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd"}
Sep 15 12:18:59.004100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:59.003616    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:18:59.026568 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:18:59.026505    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" podStartSLOduration=2.564436311 podStartE2EDuration="3.026487562s" totalImagesPullingTime="462.051251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:59.024759113 +0000 UTC m=+1019.721318702" watchObservedRunningTime="2026-09-15 12:18:59.026487562 +0000 UTC m=+1019.723047139"
Sep 15 12:19:02.840082 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:02.840016    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:02.955454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:02.955417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:03.097582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:03.097475    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:05.015233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:05.015203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:19:13.610454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:13.610417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:14.395500 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:14.395465    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:16.098303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:16.098272    2583 generic.go:356] "Generic (PLEG): container finished" podID="3663b0cd-e965-48d1-be03-2184715d86d3" containerID="9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812" exitCode=0
Sep 15 12:19:16.098734 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:16.098356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" event={"ID":"3663b0cd-e965-48d1-be03-2184715d86d3","Type":"ContainerDied","Data":"9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"}
Sep 15 12:19:16.098840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:16.098821    2583 scope.go:117] "RemoveContainer" containerID="9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"
Sep 15 12:19:16.099953 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:16.099926    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:19:16.100005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:16.099994    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:19:16.100061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:16.100039    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:19:17.106569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:17.106522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" event={"ID":"3663b0cd-e965-48d1-be03-2184715d86d3","Type":"ContainerStarted","Data":"e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d"}
Sep 15 12:19:17.881243 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:17.881204    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:19:17.882532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:17.882506    2583 patch_prober.go:28] interesting pod/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 15 12:19:17.882642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:17.882556    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 15 12:19:18.882175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:18.882143    2583 patch_prober.go:28] interesting pod/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 15 12:19:18.882567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:18.882197    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 15 12:19:19.724400 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:19.724340    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:19.881302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:19.881267    2583 patch_prober.go:28] interesting pod/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 15 12:19:19.881479 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:19.881322    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 15 12:19:20.578523 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:20.578474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:20.882272 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:20.882165    2583 patch_prober.go:28] interesting pod/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 15 12:19:20.882272 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:20.882238    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 15 12:19:20.954932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:20.954893    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:19:21.746940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:21.746896    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:21.881761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:21.881727    2583 patch_prober.go:28] interesting pod/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 15 12:19:21.881927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:21.881780    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 15 12:19:22.070612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.070574    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:22.205478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.205440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:22.378721 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.378629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:22.477472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.477430    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:22.848018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.847976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:22.886009 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.885983    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:19:22.887170 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:22.887146    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:19:23.120794 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:23.120703    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:23.250875 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:23.250839    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:24.024991 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:24.024957    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:24.824759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:24.824726    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:24.936570 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:24.936505    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:29.565247 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:29.565190    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:29.664513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:29.664462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:31.702600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:31.702557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:31.804629 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:31.804587    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:32.365884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:32.365851    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:39.565384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:39.565348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:39.665861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:39.665818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:39.824715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:39.824630    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:40.343000 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:40.342954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:40.563386 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:40.563356    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:47.181899 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:47.181840    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:47.335154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:47.335112    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:48.647975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:48.647935    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:49.869353 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:49.869302    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:50.866514 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:50.866481    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:51.912014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:51.911904    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:52.735954 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.735919    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"]
Sep 15 12:19:52.739318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.739294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.743644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.743619    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-167943df-serving-cert\""
Sep 15 12:19:52.755048 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.754985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"]
Sep 15 12:19:52.792261 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.792226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-metrics-certs\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.792431 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.792358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.792431 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.792398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdv5\" (UniqueName: \"kubernetes.io/projected/5db0a82c-67c3-4b80-a543-a1d28172c543-kube-api-access-2fdv5\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.792554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.792457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db0a82c-67c3-4b80-a543-a1d28172c543-openshift-service-ca\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.806403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.806361    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:52.893745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.893710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db0a82c-67c3-4b80-a543-a1d28172c543-openshift-service-ca\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.893918 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.893780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-metrics-certs\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.893918 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.893892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.894052 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.893920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdv5\" (UniqueName: \"kubernetes.io/projected/5db0a82c-67c3-4b80-a543-a1d28172c543-kube-api-access-2fdv5\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.894052 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:19:52.894017    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-167943df-serving-cert: secret "maas-api-e2e-mt-167943df-serving-cert" not found
Sep 15 12:19:52.894167 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:19:52.894109    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls podName:5db0a82c-67c3-4b80-a543-a1d28172c543 nodeName:}" failed. No retries permitted until 2026-09-15 12:19:53.394088502 +0000 UTC m=+1074.090648073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls") pod "maas-api-e2e-mt-167943df-75856f4b57-9rqff" (UID: "5db0a82c-67c3-4b80-a543-a1d28172c543") : secret "maas-api-e2e-mt-167943df-serving-cert" not found
Sep 15 12:19:52.894442 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.894413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db0a82c-67c3-4b80-a543-a1d28172c543-openshift-service-ca\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.896505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.896477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-metrics-certs\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.911002 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.910978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdv5\" (UniqueName: \"kubernetes.io/projected/5db0a82c-67c3-4b80-a543-a1d28172c543-kube-api-access-2fdv5\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:52.983580 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:52.983537    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:53.087745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.087713    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:53.399637 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.399541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:53.402074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.402051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls\") pod \"maas-api-e2e-mt-167943df-75856f4b57-9rqff\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:53.405244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.405193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:53.497316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.497283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:53.654407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.654308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:53.951660 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:53.951633    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"]
Sep 15 12:19:53.953901 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:19:53.953868    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db0a82c_67c3_4b80_a543_a1d28172c543.slice/crio-1ae5e810332ad811ae76649cb321d6346d8669fc60a3c459a8fd99408e737e34 WatchSource:0}: Error finding container 1ae5e810332ad811ae76649cb321d6346d8669fc60a3c459a8fd99408e737e34: Status 404 returned error can't find the container with id 1ae5e810332ad811ae76649cb321d6346d8669fc60a3c459a8fd99408e737e34
Sep 15 12:19:54.326704 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:54.326661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff" event={"ID":"5db0a82c-67c3-4b80-a543-a1d28172c543","Type":"ContainerStarted","Data":"1ae5e810332ad811ae76649cb321d6346d8669fc60a3c459a8fd99408e737e34"}
Sep 15 12:19:55.206984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:55.206943    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:19:55.334324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:55.334289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff" event={"ID":"5db0a82c-67c3-4b80-a543-a1d28172c543","Type":"ContainerStarted","Data":"3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43"}
Sep 15 12:19:55.334825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:55.334395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:19:55.355907 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:19:55.355856    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff" podStartSLOduration=2.84990063 podStartE2EDuration="3.355840103s" totalImagesPullingTime="505.939473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:19:55.354704481 +0000 UTC m=+1076.051264072" watchObservedRunningTime="2026-09-15 12:19:55.355840103 +0000 UTC m=+1076.052399681"
Sep 15 12:20:01.345314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:01.345279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:20:03.050621 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:03.050584    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:13.977304 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:13.977263    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:16.618293 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:16.618247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:16.719297 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:16.719261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:16.804699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:16.804660    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:17.130680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:17.130649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:17.227964 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:17.227931    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:17.848228 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:17.848193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:17.972786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:17.972753    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:18.251419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:18.251321    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:18.580183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:18.580144    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:19.532955 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:19.532910    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:20.324517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:20.324481    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:21.424734 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:21.424702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:22.245591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:22.245555    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:23.497946 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:23.497905    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:24.550809 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:24.550774    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:25.529778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:25.529746    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:25.668500 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:25.668455    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:26.036500 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:26.036460    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:26.933038 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:26.932998    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:27.640521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:27.640490    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"]
Sep 15 12:20:27.640701 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:27.640671    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" podUID="e00eed17-42bc-4585-90d5-2730249314dc" containerName="maas-api" containerID="cri-o://99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd" gracePeriod=30
Sep 15 12:20:27.686355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:27.686297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:27.791244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:27.791210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:27.902101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:27.902067    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:27.917107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:27.917069    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:20:28.006943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.006869    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:28.045322 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.045286    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00eed17-42bc-4585-90d5-2730249314dc-openshift-service-ca\") pod \"e00eed17-42bc-4585-90d5-2730249314dc\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") "
Sep 15 12:20:28.045503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.045371    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls\") pod \"e00eed17-42bc-4585-90d5-2730249314dc\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") "
Sep 15 12:20:28.045503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.045400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-metrics-certs\") pod \"e00eed17-42bc-4585-90d5-2730249314dc\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") "
Sep 15 12:20:28.045503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.045421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppzx2\" (UniqueName: \"kubernetes.io/projected/e00eed17-42bc-4585-90d5-2730249314dc-kube-api-access-ppzx2\") pod \"e00eed17-42bc-4585-90d5-2730249314dc\" (UID: \"e00eed17-42bc-4585-90d5-2730249314dc\") "
Sep 15 12:20:28.045764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.045732    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00eed17-42bc-4585-90d5-2730249314dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e00eed17-42bc-4585-90d5-2730249314dc" (UID: "e00eed17-42bc-4585-90d5-2730249314dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:20:28.047656 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.047622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e00eed17-42bc-4585-90d5-2730249314dc" (UID: "e00eed17-42bc-4585-90d5-2730249314dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:20:28.047758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.047671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e00eed17-42bc-4585-90d5-2730249314dc" (UID: "e00eed17-42bc-4585-90d5-2730249314dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:20:28.047758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.047707    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00eed17-42bc-4585-90d5-2730249314dc-kube-api-access-ppzx2" (OuterVolumeSpecName: "kube-api-access-ppzx2") pod "e00eed17-42bc-4585-90d5-2730249314dc" (UID: "e00eed17-42bc-4585-90d5-2730249314dc"). InnerVolumeSpecName "kube-api-access-ppzx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:28.113056 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.112944    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:28.146825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.146718    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e00eed17-42bc-4585-90d5-2730249314dc-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.146825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.146764    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.146825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.146779    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e00eed17-42bc-4585-90d5-2730249314dc-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.146825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.146792    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppzx2\" (UniqueName: \"kubernetes.io/projected/e00eed17-42bc-4585-90d5-2730249314dc-kube-api-access-ppzx2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:28.377266 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.377226    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:28.518176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.518075    2583 generic.go:356] "Generic (PLEG): container finished" podID="e00eed17-42bc-4585-90d5-2730249314dc" containerID="99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd" exitCode=0
Sep 15 12:20:28.518176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.518125    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"
Sep 15 12:20:28.518425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.518165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" event={"ID":"e00eed17-42bc-4585-90d5-2730249314dc","Type":"ContainerDied","Data":"99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd"}
Sep 15 12:20:28.518425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.518258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq" event={"ID":"e00eed17-42bc-4585-90d5-2730249314dc","Type":"ContainerDied","Data":"c5416078ebc43ae483f4f49a5497e3d8775880f35647d6df61e892d9496c691b"}
Sep 15 12:20:28.518425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.518289    2583 scope.go:117] "RemoveContainer" containerID="99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd"
Sep 15 12:20:28.527974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.527950    2583 scope.go:117] "RemoveContainer" containerID="99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd"
Sep 15 12:20:28.528317 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:20:28.528287    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd\": container with ID starting with 99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd not found: ID does not exist" containerID="99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd"
Sep 15 12:20:28.528400 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.528327    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd"} err="failed to get container status \"99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd\": rpc error: code = NotFound desc = could not find container \"99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd\": container with ID starting with 99d2ecdc8db8b89fc868b5fb3fad168f167b68c2badfe9e551236e11b06b4bbd not found: ID does not exist"
Sep 15 12:20:28.543642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.543612    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"]
Sep 15 12:20:28.552556 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:28.552528    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-294781-5cc78cd9bf-ps6rq"]
Sep 15 12:20:29.176115 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:29.176070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:29.808177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:29.808146    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00eed17-42bc-4585-90d5-2730249314dc" path="/var/lib/kubelet/pods/e00eed17-42bc-4585-90d5-2730249314dc/volumes"
Sep 15 12:20:30.240376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:30.240342    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:31.113874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:31.113838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:32.101908 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:32.101873    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:33.033834 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:33.033803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:33.957511 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:33.957474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:35.279394 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:35.279349    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:35.383668 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:35.383629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:35.624949 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:35.624911    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"]
Sep 15 12:20:35.625211 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:35.625185    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" containerID="cri-o://e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d" gracePeriod=30
Sep 15 12:20:35.643710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:35.643674    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:36.023439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:36.023348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:36.128905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:36.128865    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:36.412755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:36.412705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:36.802637 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:36.802604    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"]
Sep 15 12:20:36.802839 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:36.802810    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff" podUID="5db0a82c-67c3-4b80-a543-a1d28172c543" containerName="maas-api" containerID="cri-o://3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43" gracePeriod=30
Sep 15 12:20:36.854658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:36.854628    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:37.021893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.021789    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:37.074368 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.074344    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:20:37.134160 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.134130    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fdv5\" (UniqueName: \"kubernetes.io/projected/5db0a82c-67c3-4b80-a543-a1d28172c543-kube-api-access-2fdv5\") pod \"5db0a82c-67c3-4b80-a543-a1d28172c543\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") "
Sep 15 12:20:37.134363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.134183    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-metrics-certs\") pod \"5db0a82c-67c3-4b80-a543-a1d28172c543\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") "
Sep 15 12:20:37.134363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.134219    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db0a82c-67c3-4b80-a543-a1d28172c543-openshift-service-ca\") pod \"5db0a82c-67c3-4b80-a543-a1d28172c543\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") "
Sep 15 12:20:37.134363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.134277    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls\") pod \"5db0a82c-67c3-4b80-a543-a1d28172c543\" (UID: \"5db0a82c-67c3-4b80-a543-a1d28172c543\") "
Sep 15 12:20:37.134707 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.134672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db0a82c-67c3-4b80-a543-a1d28172c543-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5db0a82c-67c3-4b80-a543-a1d28172c543" (UID: "5db0a82c-67c3-4b80-a543-a1d28172c543"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:20:37.136638 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.136594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5db0a82c-67c3-4b80-a543-a1d28172c543" (UID: "5db0a82c-67c3-4b80-a543-a1d28172c543"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:20:37.136748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.136659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db0a82c-67c3-4b80-a543-a1d28172c543-kube-api-access-2fdv5" (OuterVolumeSpecName: "kube-api-access-2fdv5") pod "5db0a82c-67c3-4b80-a543-a1d28172c543" (UID: "5db0a82c-67c3-4b80-a543-a1d28172c543"). InnerVolumeSpecName "kube-api-access-2fdv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:37.136798 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.136762    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5db0a82c-67c3-4b80-a543-a1d28172c543" (UID: "5db0a82c-67c3-4b80-a543-a1d28172c543"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:20:37.179790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.179750    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:37.235587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.235556    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:37.235587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.235584    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fdv5\" (UniqueName: \"kubernetes.io/projected/5db0a82c-67c3-4b80-a543-a1d28172c543-kube-api-access-2fdv5\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:37.235587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.235594    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db0a82c-67c3-4b80-a543-a1d28172c543-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:37.235818 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.235603    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db0a82c-67c3-4b80-a543-a1d28172c543-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:37.279635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.279528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:37.574642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.574606    2583 generic.go:356] "Generic (PLEG): container finished" podID="5db0a82c-67c3-4b80-a543-a1d28172c543" containerID="3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43" exitCode=0
Sep 15 12:20:37.575064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.574653    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"
Sep 15 12:20:37.575064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.574662    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff" event={"ID":"5db0a82c-67c3-4b80-a543-a1d28172c543","Type":"ContainerDied","Data":"3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43"}
Sep 15 12:20:37.575064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.574722    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff" event={"ID":"5db0a82c-67c3-4b80-a543-a1d28172c543","Type":"ContainerDied","Data":"1ae5e810332ad811ae76649cb321d6346d8669fc60a3c459a8fd99408e737e34"}
Sep 15 12:20:37.575064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.574748    2583 scope.go:117] "RemoveContainer" containerID="3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43"
Sep 15 12:20:37.584800 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.584782    2583 scope.go:117] "RemoveContainer" containerID="3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43"
Sep 15 12:20:37.585097 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:20:37.585067    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43\": container with ID starting with 3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43 not found: ID does not exist" containerID="3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43"
Sep 15 12:20:37.585148 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.585106    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43"} err="failed to get container status \"3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43\": rpc error: code = NotFound desc = could not find container \"3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43\": container with ID starting with 3c70053281fe109e2c80f1bee0d8394c8d4862904f93744e6a083b6f2f276a43 not found: ID does not exist"
Sep 15 12:20:37.599090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.599063    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"]
Sep 15 12:20:37.608611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.608582    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-167943df-75856f4b57-9rqff"]
Sep 15 12:20:37.635200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.635169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:37.717069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.717005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:37.807751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.807716    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db0a82c-67c3-4b80-a543-a1d28172c543" path="/var/lib/kubelet/pods/5db0a82c-67c3-4b80-a543-a1d28172c543/volumes"
Sep 15 12:20:37.888912 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.888823    2583 patch_prober.go:28] interesting pod/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:20:37.888912 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.888900    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:20:37.945254 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:37.945202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:38.120460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:38.120420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:38.274351 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:38.274250    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:39.616279 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:39.616245    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:40.536972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.536915    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:40.894654 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.894628    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:20:40.969875 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.969839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-istio-token\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.969910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-credential-socket\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.969977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-certs\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.969995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-data\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3663b0cd-e965-48d1-be03-2184715d86d3-istio-podinfo\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3663b0cd-e965-48d1-be03-2184715d86d3-istiod-ca-cert\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj87h\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-kube-api-access-lj87h\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970123    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-envoy\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970149    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-socket\") pod \"3663b0cd-e965-48d1-be03-2184715d86d3\" (UID: \"3663b0cd-e965-48d1-be03-2184715d86d3\") "
Sep 15 12:20:40.970426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970308    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:40.970426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970338    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:40.970426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970325    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:40.970634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970453    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:40.970634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970472    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:40.970634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3663b0cd-e965-48d1-be03-2184715d86d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:20:40.970634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970486    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:40.970634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.970558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:40.972190 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.972161    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3663b0cd-e965-48d1-be03-2184715d86d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:20:40.972292 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.972270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:20:40.972378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.972355    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:40.972431 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:40.972390    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-kube-api-access-lj87h" (OuterVolumeSpecName: "kube-api-access-lj87h") pod "3663b0cd-e965-48d1-be03-2184715d86d3" (UID: "3663b0cd-e965-48d1-be03-2184715d86d3"). InnerVolumeSpecName "kube-api-access-lj87h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:20:41.071965 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.071930    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:41.071965 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.071962    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3663b0cd-e965-48d1-be03-2184715d86d3-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:41.071965 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.071972    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3663b0cd-e965-48d1-be03-2184715d86d3-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:41.072229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.071980    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj87h\" (UniqueName: \"kubernetes.io/projected/3663b0cd-e965-48d1-be03-2184715d86d3-kube-api-access-lj87h\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:41.072229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.071991    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:41.072229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.071999    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3663b0cd-e965-48d1-be03-2184715d86d3-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:20:41.319994 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.319953    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:41.602403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.602371    2583 generic.go:356] "Generic (PLEG): container finished" podID="3663b0cd-e965-48d1-be03-2184715d86d3" containerID="e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d" exitCode=0
Sep 15 12:20:41.602567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.602427    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"
Sep 15 12:20:41.602567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.602443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" event={"ID":"3663b0cd-e965-48d1-be03-2184715d86d3","Type":"ContainerDied","Data":"e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d"}
Sep 15 12:20:41.602567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.602484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw" event={"ID":"3663b0cd-e965-48d1-be03-2184715d86d3","Type":"ContainerDied","Data":"cef02ab2c7110852508b8ded957854e0b9d7934d368254690d5e41fde0f6e88a"}
Sep 15 12:20:41.602567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.602506    2583 scope.go:117] "RemoveContainer" containerID="e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d"
Sep 15 12:20:41.613540 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.613517    2583 scope.go:117] "RemoveContainer" containerID="9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"
Sep 15 12:20:41.622774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.622756    2583 scope.go:117] "RemoveContainer" containerID="e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d"
Sep 15 12:20:41.623062 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:20:41.623043    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d\": container with ID starting with e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d not found: ID does not exist" containerID="e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d"
Sep 15 12:20:41.623136 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.623070    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d"} err="failed to get container status \"e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d\": rpc error: code = NotFound desc = could not find container \"e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d\": container with ID starting with e0ab2d19c2083b4b1a10a8b455e995e12073734e8ca0df7dc45cfa0aab95c28d not found: ID does not exist"
Sep 15 12:20:41.623136 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.623086    2583 scope.go:117] "RemoveContainer" containerID="9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"
Sep 15 12:20:41.623336 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:20:41.623317    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812\": container with ID starting with 9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812 not found: ID does not exist" containerID="9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"
Sep 15 12:20:41.623376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.623343    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812"} err="failed to get container status \"9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812\": rpc error: code = NotFound desc = could not find container \"9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812\": container with ID starting with 9e8340d1ebc2409916c503ab2c187bfaac9d67d8ae849b5d9efca4b270896812 not found: ID does not exist"
Sep 15 12:20:41.628875 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.628852    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"]
Sep 15 12:20:41.635153 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.635130    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-294781-openshift-default-8774d4c84-wkjrw"]
Sep 15 12:20:41.807059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:41.807005    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" path="/var/lib/kubelet/pods/3663b0cd-e965-48d1-be03-2184715d86d3/volumes"
Sep 15 12:20:42.231066 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:42.231000    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:44.800151 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:44.800111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:44.899306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:44.899271    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:45.083113 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.083075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:45.418567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.418474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:45.645299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.645266    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:45.742137 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742047    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"]
Sep 15 12:20:45.742624 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742606    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db0a82c-67c3-4b80-a543-a1d28172c543" containerName="maas-api"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742627    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db0a82c-67c3-4b80-a543-a1d28172c543" containerName="maas-api"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742656    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742663    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742670    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00eed17-42bc-4585-90d5-2730249314dc" containerName="maas-api"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742675    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00eed17-42bc-4585-90d5-2730249314dc" containerName="maas-api"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742683    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy"
Sep 15 12:20:45.742726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742688    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy"
Sep 15 12:20:45.743060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742758    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00eed17-42bc-4585-90d5-2730249314dc" containerName="maas-api"
Sep 15 12:20:45.743060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742768    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db0a82c-67c3-4b80-a543-a1d28172c543" containerName="maas-api"
Sep 15 12:20:45.743060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742774    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy"
Sep 15 12:20:45.743060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.742782    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3663b0cd-e965-48d1-be03-2184715d86d3" containerName="istio-proxy"
Sep 15 12:20:45.747441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.747422    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.751856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.751832    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"openshift-service-ca.crt\""
Sep 15 12:20:45.751856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.751848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"default-dockercfg-mgxdh\""
Sep 15 12:20:45.752011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.751848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"e2e-conflict-a-a7f06f2e-kserve-self-signed-certs\""
Sep 15 12:20:45.752011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.751852    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9ba48f\"/\"kube-root-ca.crt\""
Sep 15 12:20:45.765486 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.765459    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"]
Sep 15 12:20:45.803071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.803004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:45.815513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.815477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tmp-dir\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.815692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.815590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tls-certs\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.815760 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.815703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-model-cache\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.815760 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.815745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6s2\" (UniqueName: \"kubernetes.io/projected/5777ecd8-a48b-4a45-9dc8-5c75aad99367-kube-api-access-5t6s2\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.815932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.815824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-dshm\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.815932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.815857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-home\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.903324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.903287    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:45.916783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.916752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tls-certs\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.916972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.916811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-model-cache\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.916972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.916844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6s2\" (UniqueName: \"kubernetes.io/projected/5777ecd8-a48b-4a45-9dc8-5c75aad99367-kube-api-access-5t6s2\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.916972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.916885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-dshm\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.916972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.916911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-home\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.916972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.916937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tmp-dir\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.917384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.917359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-model-cache\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.917495 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.917376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tmp-dir\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.917600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.917572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-home\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.919446 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.919425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tls-certs\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.919545 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.919515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-dshm\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:45.928158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:45.928128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6s2\" (UniqueName: \"kubernetes.io/projected/5777ecd8-a48b-4a45-9dc8-5c75aad99367-kube-api-access-5t6s2\") pod \"e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") " pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:46.056730 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:46.056635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:46.346959 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:46.346925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"]
Sep 15 12:20:46.350295 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:20:46.350252    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5777ecd8_a48b_4a45_9dc8_5c75aad99367.slice/crio-901dda685312f8ab0992ec063061f1c882246f253fbc1d27d61e8771aebde485 WatchSource:0}: Error finding container 901dda685312f8ab0992ec063061f1c882246f253fbc1d27d61e8771aebde485: Status 404 returned error can't find the container with id 901dda685312f8ab0992ec063061f1c882246f253fbc1d27d61e8771aebde485
Sep 15 12:20:46.634336 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:46.634244    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr" event={"ID":"5777ecd8-a48b-4a45-9dc8-5c75aad99367","Type":"ContainerStarted","Data":"7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987"}
Sep 15 12:20:46.634336 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:46.634294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr" event={"ID":"5777ecd8-a48b-4a45-9dc8-5c75aad99367","Type":"ContainerStarted","Data":"901dda685312f8ab0992ec063061f1c882246f253fbc1d27d61e8771aebde485"}
Sep 15 12:20:46.667962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:46.667901    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr" podStartSLOduration=1.667884011 podStartE2EDuration="1.667884011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:20:46.667557649 +0000 UTC m=+1127.364117226" watchObservedRunningTime="2026-09-15 12:20:46.667884011 +0000 UTC m=+1127.364443589"
Sep 15 12:20:47.916598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:47.916556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:49.495182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:49.495146    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:52.599661 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:52.599620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:53.186839 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:53.186792    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:53.319466 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:53.319432    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:53.429697 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:53.429661    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:53.595037 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:53.594985    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:20:56.057098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:56.057065    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:56.057521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:56.057218    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:56.069764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:56.069733    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:56.699731 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:56.699693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:20:59.977846 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:20:59.977809    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:00.075407 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:00.075357    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:00.886772 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:00.886730    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:01.202550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.202462    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"]
Sep 15 12:21:01.207810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.207783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.211618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.211590    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-58b0fd-openshift-default-dockercfg-cvdrd\""
Sep 15 12:21:01.222538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.222509    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"]
Sep 15 12:21:01.239656 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.239605    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:01.268215 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-data\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268381 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268381 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-certs\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268455 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-podinfo\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268455 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-socket\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268455 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-envoy\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-credential-socket\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268583 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9l2\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-kube-api-access-xr9l2\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.268583 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.268577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-token\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370123 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-certs\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-podinfo\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-socket\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-envoy\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-credential-socket\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9l2\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-kube-api-access-xr9l2\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-token\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-certs\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-data\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-socket\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370847 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-credential-socket\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.370969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.370941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-data\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.371270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.371252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.372558 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.372532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-envoy\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.372899 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.372880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-podinfo\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.380696 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.380673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-token\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.380820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.380799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9l2\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-kube-api-access-xr9l2\") pod \"e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") " pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.519667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.519573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:01.553139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.553111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:01.804179 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:21:01.804141    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b54dcf7_1f29_49bd_bfd5_2650deeccda7.slice/crio-aab11b9f57b86fc0aa571af7a7df20adba8af307464e1690420779aa4835868b WatchSource:0}: Error finding container aab11b9f57b86fc0aa571af7a7df20adba8af307464e1690420779aa4835868b: Status 404 returned error can't find the container with id aab11b9f57b86fc0aa571af7a7df20adba8af307464e1690420779aa4835868b
Sep 15 12:21:01.807119 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.807076    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:21:01.807226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.807166    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:21:01.807226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.807208    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:21:01.809325 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:01.809305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"]
Sep 15 12:21:02.722978 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:02.722944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" event={"ID":"3b54dcf7-1f29-49bd-bfd5-2650deeccda7","Type":"ContainerStarted","Data":"e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"}
Sep 15 12:21:02.723603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:02.723008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" event={"ID":"3b54dcf7-1f29-49bd-bfd5-2650deeccda7","Type":"ContainerStarted","Data":"aab11b9f57b86fc0aa571af7a7df20adba8af307464e1690420779aa4835868b"}
Sep 15 12:21:02.754059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:02.753989    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podStartSLOduration=1.753971735 podStartE2EDuration="1.753971735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:02.752079656 +0000 UTC m=+1143.448639245" watchObservedRunningTime="2026-09-15 12:21:02.753971735 +0000 UTC m=+1143.450531313"
Sep 15 12:21:03.519927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:03.519888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:03.526237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:03.526210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:03.730312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:03.730277    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:03.731379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:03.731358    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:04.177714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:04.177683    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:04.682947 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:04.682911    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:04.796797 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:04.796760    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:05.285634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:05.285600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:06.206263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:06.206221    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:07.176085 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:07.176050    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:08.239642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:08.239602    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:09.179910 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.179870    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"]
Sep 15 12:21:09.184617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.184591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.194924 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.194896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-58b0fd-serving-cert\""
Sep 15 12:21:09.208623 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.208578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"]
Sep 15 12:21:09.255556 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.255516    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:09.357316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.357279    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:09.358345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.358322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.358457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.358436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.358500 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.358464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.358536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.358520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hng\" (UniqueName: \"kubernetes.io/projected/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-kube-api-access-69hng\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.456399 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.456305    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:09.459481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.459453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.459640 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.459515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.459640 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.459537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.459640 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.459557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hng\" (UniqueName: \"kubernetes.io/projected/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-kube-api-access-69hng\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.460323 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.460303    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.462102 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.462080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.462277 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.462256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.468338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.468318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hng\" (UniqueName: \"kubernetes.io/projected/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-kube-api-access-69hng\") pod \"maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.496394 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.496350    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:09.808624 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:21:09.808589    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb3fea2_df4e_472f_9f34_11ec2a62bbfa.slice/crio-051f53513d3e56ee96c8ec10ec0cc8785afe6f83c63fffa96a0cb20e67037a1c WatchSource:0}: Error finding container 051f53513d3e56ee96c8ec10ec0cc8785afe6f83c63fffa96a0cb20e67037a1c: Status 404 returned error can't find the container with id 051f53513d3e56ee96c8ec10ec0cc8785afe6f83c63fffa96a0cb20e67037a1c
Sep 15 12:21:09.809662 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.809638    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"]
Sep 15 12:21:09.810289 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.810272    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:21:09.996288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:09.996255    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:10.771887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:10.771844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt" event={"ID":"deb3fea2-df4e-472f-9f34-11ec2a62bbfa","Type":"ContainerStarted","Data":"a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6"}
Sep 15 12:21:10.772319 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:10.771908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt" event={"ID":"deb3fea2-df4e-472f-9f34-11ec2a62bbfa","Type":"ContainerStarted","Data":"051f53513d3e56ee96c8ec10ec0cc8785afe6f83c63fffa96a0cb20e67037a1c"}
Sep 15 12:21:10.772319 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:10.772049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:10.794425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:10.794325    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt" podStartSLOduration=1.070250061 podStartE2EDuration="1.794310159s" totalImagesPullingTime="724.060098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:10.792307639 +0000 UTC m=+1151.488867216" watchObservedRunningTime="2026-09-15 12:21:10.794310159 +0000 UTC m=+1151.490869737"
Sep 15 12:21:11.031502 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:11.031463    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:11.131838 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:11.131796    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:11.458491 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:11.458397    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:16.783185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:16.783141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:21:30.564999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:30.564965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:32.060617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:32.060582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:34.171111 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:34.171075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:34.273456 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:34.273414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:34.759131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:34.759093    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:34.853835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:34.853801    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:35.152353 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:35.152316    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:35.557887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:35.557805    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:35.645691 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:35.645653    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:36.041753 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:36.041703    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:37.037582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:37.037497    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:37.845957 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:37.845917    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:37.960602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:37.960562    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:38.058272 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:38.058197    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:38.573759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:38.573719    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:38.682366 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:38.682325    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:39.131401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:39.131357    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:40.235543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.235421    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:40.944300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.944270    2583 generic.go:356] "Generic (PLEG): container finished" podID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerID="e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40" exitCode=0
Sep 15 12:21:40.944465 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.944337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" event={"ID":"3b54dcf7-1f29-49bd-bfd5-2650deeccda7","Type":"ContainerDied","Data":"e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"}
Sep 15 12:21:40.944812 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.944795    2583 scope.go:117] "RemoveContainer" containerID="e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"
Sep 15 12:21:40.946108 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.946079    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:21:40.946182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.946149    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:21:40.946227 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:40.946198    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:21:41.000379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:41.000338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:41.951925 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:41.951892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" event={"ID":"3b54dcf7-1f29-49bd-bfd5-2650deeccda7","Type":"ContainerStarted","Data":"35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53"}
Sep 15 12:21:42.048375 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:42.048335    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:42.520521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:42.520489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:42.520913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:42.520888    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 12:21:42.521109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:42.521082    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 12:21:42.889895 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:42.889859    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:43.520881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.520845    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 12:21:43.521498 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.521467    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 12:21:43.837638 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.837608    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"]
Sep 15 12:21:43.850668 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.848297    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:43.850668 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.849609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"]
Sep 15 12:21:43.855463 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.855437    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ba366c-serving-cert\""
Sep 15 12:21:43.878710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.878672    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:43.977953 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:43.977917    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:44.033586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.033542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zp9m\" (UniqueName: \"kubernetes.io/projected/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-kube-api-access-9zp9m\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.033790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.033638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.033790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.033683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.033790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.033738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.136053 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.135912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.136053 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.135009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.136242 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.136149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp9m\" (UniqueName: \"kubernetes.io/projected/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-kube-api-access-9zp9m\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.136242 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.136216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.136318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.136250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.136909 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:21:44.136460    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-serving-cert: secret "maas-api-e2e-shared-b-w2-ba366c-serving-cert" not found
Sep 15 12:21:44.137098 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:21:44.137077    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls podName:d2d79c4b-25bb-4f4b-b6b5-697e9331236a nodeName:}" failed. No retries permitted until 2026-09-15 12:21:44.637053988 +0000 UTC m=+1185.333613555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" (UID: "d2d79c4b-25bb-4f4b-b6b5-697e9331236a") : secret "maas-api-e2e-shared-b-w2-ba366c-serving-cert" not found
Sep 15 12:21:44.138675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.138655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.145983 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.145960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp9m\" (UniqueName: \"kubernetes.io/projected/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-kube-api-access-9zp9m\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.254425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.254386    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:44.445525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.445417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:44.520441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.520405    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 12:21:44.520707 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.520674    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 12:21:44.552008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.551959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:44.641300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.641260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.643837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.643813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:44.763714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:44.763624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:45.074828 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.074792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"]
Sep 15 12:21:45.078517 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:21:45.078487    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d79c4b_25bb_4f4b_b6b5_697e9331236a.slice/crio-4b612e9cb0aade459d4fb4780b8802d2cd92333dc37a7f233f24b54b8d7e300f WatchSource:0}: Error finding container 4b612e9cb0aade459d4fb4780b8802d2cd92333dc37a7f233f24b54b8d7e300f: Status 404 returned error can't find the container with id 4b612e9cb0aade459d4fb4780b8802d2cd92333dc37a7f233f24b54b8d7e300f
Sep 15 12:21:45.521251 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.520708    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 12:21:45.521251 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.520776    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 12:21:45.976516 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.976468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" event={"ID":"d2d79c4b-25bb-4f4b-b6b5-697e9331236a","Type":"ContainerStarted","Data":"11b04ad1e2e4d55b63c11ab527f9ef09afc484691138df12f14992da366f4da2"}
Sep 15 12:21:45.976871 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.976540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" event={"ID":"d2d79c4b-25bb-4f4b-b6b5-697e9331236a","Type":"ContainerStarted","Data":"4b612e9cb0aade459d4fb4780b8802d2cd92333dc37a7f233f24b54b8d7e300f"}
Sep 15 12:21:45.976871 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.976650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:45.998555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:45.998509    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" podStartSLOduration=2.532352343 podStartE2EDuration="2.998493552s" totalImagesPullingTime="466.141209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:21:45.996433604 +0000 UTC m=+1186.692993204" watchObservedRunningTime="2026-09-15 12:21:45.998493552 +0000 UTC m=+1186.695053130"
Sep 15 12:21:46.520673 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:46.520625    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 12:21:46.520892 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:46.520707    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 12:21:47.525296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:47.525266    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:47.525707 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:47.525604    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:47.526246 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:47.526227    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:21:51.988000 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:51.987968    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:21:56.502524 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:56.502489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:21:59.815682 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:59.815651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:21:59.816775 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:21:59.816748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:22:00.362392 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:00.362340    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:01.394542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:01.394490    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:01.622311 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:01.622268    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:01.772363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:01.772278    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:03.631894 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:03.631854    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:04.217650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:04.217611    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:04.473671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:04.473577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:05.356356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:05.356316    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:06.138147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:06.138116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:06.799049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:06.798991    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:07.246881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:07.246848    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:08.088922 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:08.088884    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:09.068316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:09.068270    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:10.005911 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:10.005862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:10.935526 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:10.935489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:11.903676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:11.903639    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:11.986082 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:11.986036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:12.887684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:12.887652    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:13.593210 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:13.593173    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:13.703310 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:13.703271    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:13.814142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:13.814107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:14.323344 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.323306    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-9ee977"]
Sep 15 12:22:14.327408 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.327386    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:14.329774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.329757    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q76n5\""
Sep 15 12:22:14.335525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.335504    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-9ee977"]
Sep 15 12:22:14.446101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.446065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w648z\" (UniqueName: \"kubernetes.io/projected/f2b0bce6-5660-4031-a11f-5d32a80b4969-kube-api-access-w648z\") pod \"test-curl-7938-9ee977\" (UID: \"f2b0bce6-5660-4031-a11f-5d32a80b4969\") " pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:14.547579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.547539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w648z\" (UniqueName: \"kubernetes.io/projected/f2b0bce6-5660-4031-a11f-5d32a80b4969-kube-api-access-w648z\") pod \"test-curl-7938-9ee977\" (UID: \"f2b0bce6-5660-4031-a11f-5d32a80b4969\") " pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:14.559253 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.559223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w648z\" (UniqueName: \"kubernetes.io/projected/f2b0bce6-5660-4031-a11f-5d32a80b4969-kube-api-access-w648z\") pod \"test-curl-7938-9ee977\" (UID: \"f2b0bce6-5660-4031-a11f-5d32a80b4969\") " pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:14.598132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.598050    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:14.637051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.636990    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:14.927299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:14.927273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-9ee977"]
Sep 15 12:22:14.929813 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:14.929778    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b0bce6_5660_4031_a11f_5d32a80b4969.slice/crio-45566e310604480fb278d96a751daeb1bd8777b8d2e8e8056e0458447df7e76f WatchSource:0}: Error finding container 45566e310604480fb278d96a751daeb1bd8777b8d2e8e8056e0458447df7e76f: Status 404 returned error can't find the container with id 45566e310604480fb278d96a751daeb1bd8777b8d2e8e8056e0458447df7e76f
Sep 15 12:22:15.150074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.150002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-9ee977" event={"ID":"f2b0bce6-5660-4031-a11f-5d32a80b4969","Type":"ContainerStarted","Data":"1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929"}
Sep 15 12:22:15.150074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.150077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-9ee977" event={"ID":"f2b0bce6-5660-4031-a11f-5d32a80b4969","Type":"ContainerStarted","Data":"45566e310604480fb278d96a751daeb1bd8777b8d2e8e8056e0458447df7e76f"}
Sep 15 12:22:15.167125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.167058    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-9ee977" podStartSLOduration=1.16701724 podStartE2EDuration="1.16701724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:15.1662493 +0000 UTC m=+1215.862808877" watchObservedRunningTime="2026-09-15 12:22:15.16701724 +0000 UTC m=+1215.863576857"
Sep 15 12:22:15.625748 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.625705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:15.761835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.761802    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-9ee977"]
Sep 15 12:22:15.771274 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.771239    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-9ee977"]
Sep 15 12:22:15.879510 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.879434    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-074b38"]
Sep 15 12:22:15.879929 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.879916    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b0bce6-5660-4031-a11f-5d32a80b4969" containerName="test-curl-7938-9ee977"
Sep 15 12:22:15.879977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.879931    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b0bce6-5660-4031-a11f-5d32a80b4969" containerName="test-curl-7938-9ee977"
Sep 15 12:22:15.880013 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.880004    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b0bce6-5660-4031-a11f-5d32a80b4969" containerName="test-curl-7938-9ee977"
Sep 15 12:22:15.883250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.883229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:15.891861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.891836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-074b38"]
Sep 15 12:22:15.962444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:15.962410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hbkq\" (UniqueName: \"kubernetes.io/projected/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe-kube-api-access-5hbkq\") pod \"test-curl-7938-074b38\" (UID: \"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe\") " pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:16.063529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.063493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hbkq\" (UniqueName: \"kubernetes.io/projected/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe-kube-api-access-5hbkq\") pod \"test-curl-7938-074b38\" (UID: \"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe\") " pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:16.072193 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.072169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hbkq\" (UniqueName: \"kubernetes.io/projected/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe-kube-api-access-5hbkq\") pod \"test-curl-7938-074b38\" (UID: \"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe\") " pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:16.155883 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:16.155805    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-9ee977" container="test-curl-7938-9ee977"
Sep 15 12:22:16.193454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.193422    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:16.361420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.361387    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"]
Sep 15 12:22:16.361596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.361560    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" podUID="5f961480-9129-4b78-a061-4c9ac71ae494" containerName="maas-api" containerID="cri-o://7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074" gracePeriod=30
Sep 15 12:22:16.364981 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.364954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:16.437163 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.437055    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"]
Sep 15 12:22:16.437375 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.437343    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" podUID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" containerName="payload-processing" containerID="cri-o://7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0" gracePeriod=30
Sep 15 12:22:16.473977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.473943    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-074b38"]
Sep 15 12:22:16.487962 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:16.487931    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b59a7e_53ce_42ff_81a0_b01c116b3ebe.slice/crio-948df91834187e8efd2e89d550d570953b2617ae16c243679cee152f2822e810 WatchSource:0}: Error finding container 948df91834187e8efd2e89d550d570953b2617ae16c243679cee152f2822e810: Status 404 returned error can't find the container with id 948df91834187e8efd2e89d550d570953b2617ae16c243679cee152f2822e810
Sep 15 12:22:16.504972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.504937    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:16.622241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.622205    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:16.690198 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.690130    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:22:16.739249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.739224    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:22:16.770267 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.770237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzht8\" (UniqueName: \"kubernetes.io/projected/5f961480-9129-4b78-a061-4c9ac71ae494-kube-api-access-vzht8\") pod \"5f961480-9129-4b78-a061-4c9ac71ae494\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") "
Sep 15 12:22:16.770561 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.770294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-metrics-certs\") pod \"5f961480-9129-4b78-a061-4c9ac71ae494\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") "
Sep 15 12:22:16.770561 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.770387    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls\") pod \"5f961480-9129-4b78-a061-4c9ac71ae494\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") "
Sep 15 12:22:16.770561 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.770533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f961480-9129-4b78-a061-4c9ac71ae494-openshift-service-ca\") pod \"5f961480-9129-4b78-a061-4c9ac71ae494\" (UID: \"5f961480-9129-4b78-a061-4c9ac71ae494\") "
Sep 15 12:22:16.771130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.771061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:16.771130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.771109    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f961480-9129-4b78-a061-4c9ac71ae494-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f961480-9129-4b78-a061-4c9ac71ae494" (UID: "5f961480-9129-4b78-a061-4c9ac71ae494"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:16.784716 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.777340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f961480-9129-4b78-a061-4c9ac71ae494" (UID: "5f961480-9129-4b78-a061-4c9ac71ae494"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:16.784716 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.778471    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f961480-9129-4b78-a061-4c9ac71ae494" (UID: "5f961480-9129-4b78-a061-4c9ac71ae494"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:16.786929 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.786894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f961480-9129-4b78-a061-4c9ac71ae494-kube-api-access-vzht8" (OuterVolumeSpecName: "kube-api-access-vzht8") pod "5f961480-9129-4b78-a061-4c9ac71ae494" (UID: "5f961480-9129-4b78-a061-4c9ac71ae494"). InnerVolumeSpecName "kube-api-access-vzht8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:16.871539 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.871500    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjqhs\" (UniqueName: \"kubernetes.io/projected/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-kube-api-access-pjqhs\") pod \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") "
Sep 15 12:22:16.871885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.871862    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-plugins-config-volume\") pod \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\" (UID: \"41785fc0-b7e9-40d1-8f42-4ceaa01bd367\") "
Sep 15 12:22:16.872320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.872293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41785fc0-b7e9-40d1-8f42-4ceaa01bd367" (UID: "41785fc0-b7e9-40d1-8f42-4ceaa01bd367"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:16.872320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.872308    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f961480-9129-4b78-a061-4c9ac71ae494-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:16.872484 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.872332    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzht8\" (UniqueName: \"kubernetes.io/projected/5f961480-9129-4b78-a061-4c9ac71ae494-kube-api-access-vzht8\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:16.872484 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.872347    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:16.872484 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.872361    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f961480-9129-4b78-a061-4c9ac71ae494-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:16.874125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.874098    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-kube-api-access-pjqhs" (OuterVolumeSpecName: "kube-api-access-pjqhs") pod "41785fc0-b7e9-40d1-8f42-4ceaa01bd367" (UID: "41785fc0-b7e9-40d1-8f42-4ceaa01bd367"). InnerVolumeSpecName "kube-api-access-pjqhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:16.895422 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.895388    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:16.973820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.973732    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjqhs\" (UniqueName: \"kubernetes.io/projected/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-kube-api-access-pjqhs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:16.973820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:16.973764    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41785fc0-b7e9-40d1-8f42-4ceaa01bd367-plugins-config-volume\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:17.113177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.113100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:17.163230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.163193    2583 generic.go:356] "Generic (PLEG): container finished" podID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" containerID="7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0" exitCode=0
Sep 15 12:22:17.163384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.163255    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"
Sep 15 12:22:17.163384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.163275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" event={"ID":"41785fc0-b7e9-40d1-8f42-4ceaa01bd367","Type":"ContainerDied","Data":"7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0"}
Sep 15 12:22:17.163384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.163332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" event={"ID":"41785fc0-b7e9-40d1-8f42-4ceaa01bd367","Type":"ContainerDied","Data":"08c270a6a3a57866a9a92d0c8ed159646537716f16fc4afda817fae0be5076a2"}
Sep 15 12:22:17.163384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.163365    2583 scope.go:117] "RemoveContainer" containerID="7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0"
Sep 15 12:22:17.165067 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.165039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-074b38" event={"ID":"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe","Type":"ContainerStarted","Data":"a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707"}
Sep 15 12:22:17.165184 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.165084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-074b38" event={"ID":"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe","Type":"ContainerStarted","Data":"948df91834187e8efd2e89d550d570953b2617ae16c243679cee152f2822e810"}
Sep 15 12:22:17.166616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.166591    2583 generic.go:356] "Generic (PLEG): container finished" podID="5f961480-9129-4b78-a061-4c9ac71ae494" containerID="7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074" exitCode=0
Sep 15 12:22:17.166713 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.166648    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"
Sep 15 12:22:17.166713 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.166664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" event={"ID":"5f961480-9129-4b78-a061-4c9ac71ae494","Type":"ContainerDied","Data":"7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074"}
Sep 15 12:22:17.166840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.166712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl" event={"ID":"5f961480-9129-4b78-a061-4c9ac71ae494","Type":"ContainerDied","Data":"2da2faf1d421f04d0bd1f475919d0671c0e5f91ea586d35d20377c75d4ed41a3"}
Sep 15 12:22:17.166930 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.166905    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-9ee977" podUID="f2b0bce6-5660-4031-a11f-5d32a80b4969" containerName="test-curl-7938-9ee977" containerID="cri-o://1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929" gracePeriod=2
Sep 15 12:22:17.174181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.174165    2583 scope.go:117] "RemoveContainer" containerID="7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0"
Sep 15 12:22:17.174484 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:17.174464    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0\": container with ID starting with 7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0 not found: ID does not exist" containerID="7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0"
Sep 15 12:22:17.174542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.174493    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0"} err="failed to get container status \"7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0\": rpc error: code = NotFound desc = could not find container \"7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0\": container with ID starting with 7cf98e8a27655ad824b6b5c0e6b95b64e7265538844d0e5c09a69e4b6faa91b0 not found: ID does not exist"
Sep 15 12:22:17.174542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.174510    2583 scope.go:117] "RemoveContainer" containerID="7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074"
Sep 15 12:22:17.185781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.185756    2583 scope.go:117] "RemoveContainer" containerID="7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074"
Sep 15 12:22:17.186309 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:17.186285    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074\": container with ID starting with 7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074 not found: ID does not exist" containerID="7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074"
Sep 15 12:22:17.186433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.186320    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074"} err="failed to get container status \"7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074\": rpc error: code = NotFound desc = could not find container \"7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074\": container with ID starting with 7915fe3332c7400bc5ed5a150fa94b1c102f83f79530f8979c525bf0f9927074 not found: ID does not exist"
Sep 15 12:22:17.187621 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.187580    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-074b38" podStartSLOduration=2.187566277 podStartE2EDuration="2.187566277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:17.186454312 +0000 UTC m=+1217.883013891" watchObservedRunningTime="2026-09-15 12:22:17.187566277 +0000 UTC m=+1217.884125855"
Sep 15 12:22:17.206725 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.206691    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"]
Sep 15 12:22:17.209600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.209578    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68"]
Sep 15 12:22:17.223885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.223808    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"]
Sep 15 12:22:17.228295 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.228271    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4a9453-54bdf4b57-gstfl"]
Sep 15 12:22:17.244828 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.244797    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:17.705275 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.705223    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4a9453-6687489d45-vkm68" podUID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.52:9004: i/o timeout"
Sep 15 12:22:17.808590 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.808559    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" path="/var/lib/kubelet/pods/41785fc0-b7e9-40d1-8f42-4ceaa01bd367/volumes"
Sep 15 12:22:17.809077 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.809057    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f961480-9129-4b78-a061-4c9ac71ae494" path="/var/lib/kubelet/pods/5f961480-9129-4b78-a061-4c9ac71ae494/volumes"
Sep 15 12:22:17.990326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.990248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-074b38"]
Sep 15 12:22:17.994427 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:17.994402    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-074b38"]
Sep 15 12:22:18.103661 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.103629    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-fac56d"]
Sep 15 12:22:18.104164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104145    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f961480-9129-4b78-a061-4c9ac71ae494" containerName="maas-api"
Sep 15 12:22:18.104164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104165    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f961480-9129-4b78-a061-4c9ac71ae494" containerName="maas-api"
Sep 15 12:22:18.104291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104184    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" containerName="test-curl-7938-074b38"
Sep 15 12:22:18.104291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104190    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" containerName="test-curl-7938-074b38"
Sep 15 12:22:18.104291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104197    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" containerName="payload-processing"
Sep 15 12:22:18.104291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104202    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" containerName="payload-processing"
Sep 15 12:22:18.104291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104271    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f961480-9129-4b78-a061-4c9ac71ae494" containerName="maas-api"
Sep 15 12:22:18.104291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104284    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="41785fc0-b7e9-40d1-8f42-4ceaa01bd367" containerName="payload-processing"
Sep 15 12:22:18.104497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.104300    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" containerName="test-curl-7938-074b38"
Sep 15 12:22:18.109336 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.109312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:18.116130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.116049    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:18.118915 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.118890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-fac56d"]
Sep 15 12:22:18.174476 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:18.174442    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-074b38" container="test-curl-7938-074b38"
Sep 15 12:22:18.287283 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.287186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22r4c\" (UniqueName: \"kubernetes.io/projected/9cf39880-a473-4fc8-8b24-ef08ef18e99f-kube-api-access-22r4c\") pod \"test-curl-7938-fac56d\" (UID: \"9cf39880-a473-4fc8-8b24-ef08ef18e99f\") " pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:18.335827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.335786    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:18.388430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.388400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22r4c\" (UniqueName: \"kubernetes.io/projected/9cf39880-a473-4fc8-8b24-ef08ef18e99f-kube-api-access-22r4c\") pod \"test-curl-7938-fac56d\" (UID: \"9cf39880-a473-4fc8-8b24-ef08ef18e99f\") " pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:18.400827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.400797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22r4c\" (UniqueName: \"kubernetes.io/projected/9cf39880-a473-4fc8-8b24-ef08ef18e99f-kube-api-access-22r4c\") pod \"test-curl-7938-fac56d\" (UID: \"9cf39880-a473-4fc8-8b24-ef08ef18e99f\") " pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:18.421744 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.421719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:18.705502 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:18.705468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-fac56d"]
Sep 15 12:22:18.706857 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:18.706828    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cf39880_a473_4fc8_8b24_ef08ef18e99f.slice/crio-bb282121abe0ad759d09870bb53506af2d4edb7119a307c77496b9618748d3d0 WatchSource:0}: Error finding container bb282121abe0ad759d09870bb53506af2d4edb7119a307c77496b9618748d3d0: Status 404 returned error can't find the container with id bb282121abe0ad759d09870bb53506af2d4edb7119a307c77496b9618748d3d0
Sep 15 12:22:19.015001 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.014892    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:19.182684 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.182648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-fac56d" event={"ID":"9cf39880-a473-4fc8-8b24-ef08ef18e99f","Type":"ContainerStarted","Data":"5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d"}
Sep 15 12:22:19.182808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.182694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-fac56d" event={"ID":"9cf39880-a473-4fc8-8b24-ef08ef18e99f","Type":"ContainerStarted","Data":"bb282121abe0ad759d09870bb53506af2d4edb7119a307c77496b9618748d3d0"}
Sep 15 12:22:19.182808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.182785    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-074b38" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" containerName="test-curl-7938-074b38" containerID="cri-o://a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707" gracePeriod=2
Sep 15 12:22:19.212121 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:19.211981    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-074b38\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:19.229369 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.229331    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:19.324718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.324689    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:19.328986 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.328963    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:19.332157 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:19.332132    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-074b38\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:19.499774 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.499743    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w648z\" (UniqueName: \"kubernetes.io/projected/f2b0bce6-5660-4031-a11f-5d32a80b4969-kube-api-access-w648z\") pod \"f2b0bce6-5660-4031-a11f-5d32a80b4969\" (UID: \"f2b0bce6-5660-4031-a11f-5d32a80b4969\") "
Sep 15 12:22:19.501948 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.501922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b0bce6-5660-4031-a11f-5d32a80b4969-kube-api-access-w648z" (OuterVolumeSpecName: "kube-api-access-w648z") pod "f2b0bce6-5660-4031-a11f-5d32a80b4969" (UID: "f2b0bce6-5660-4031-a11f-5d32a80b4969"). InnerVolumeSpecName "kube-api-access-w648z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:19.601620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.601582    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w648z\" (UniqueName: \"kubernetes.io/projected/f2b0bce6-5660-4031-a11f-5d32a80b4969-kube-api-access-w648z\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:19.805361 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:19.805310    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-074b38\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:19.808174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.808148    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b0bce6-5660-4031-a11f-5d32a80b4969" path="/var/lib/kubelet/pods/f2b0bce6-5660-4031-a11f-5d32a80b4969/volumes"
Sep 15 12:22:19.850445 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.850391    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-fac56d" podStartSLOduration=1.8503712129999998 podStartE2EDuration="1.850371213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:19.209925866 +0000 UTC m=+1219.906485445" watchObservedRunningTime="2026-09-15 12:22:19.850371213 +0000 UTC m=+1220.546930790"
Sep 15 12:22:19.855155 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.855011    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-fac56d"]
Sep 15 12:22:19.860564 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.860527    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-fac56d"]
Sep 15 12:22:19.973258 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.973221    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7938-b3b8ad"]
Sep 15 12:22:19.973713 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.973699    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" containerName="test-curl-7938-fac56d"
Sep 15 12:22:19.973761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.973715    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" containerName="test-curl-7938-fac56d"
Sep 15 12:22:19.973840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.973791    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" containerName="test-curl-7938-fac56d"
Sep 15 12:22:19.977067 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.977048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:19.987401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:19.987378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-b3b8ad"]
Sep 15 12:22:20.029099 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.029053    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:20.105007 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.104974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkx9\" (UniqueName: \"kubernetes.io/projected/2039c1b8-aec2-45a6-b645-cd3af08be3e5-kube-api-access-pxkx9\") pod \"test-curl-7938-b3b8ad\" (UID: \"2039c1b8-aec2-45a6-b645-cd3af08be3e5\") " pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:20.189613 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.189523    2583 generic.go:356] "Generic (PLEG): container finished" podID="f2b0bce6-5660-4031-a11f-5d32a80b4969" containerID="1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929" exitCode=137
Sep 15 12:22:20.189613 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.189579    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-9ee977"
Sep 15 12:22:20.189828 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.189623    2583 scope.go:117] "RemoveContainer" containerID="1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929"
Sep 15 12:22:20.189888 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:20.189835    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-fac56d" container="test-curl-7938-fac56d"
Sep 15 12:22:20.198710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.198691    2583 scope.go:117] "RemoveContainer" containerID="1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929"
Sep 15 12:22:20.198972 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:20.198951    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929\": container with ID starting with 1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929 not found: ID does not exist" containerID="1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929"
Sep 15 12:22:20.199065 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.198985    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929"} err="failed to get container status \"1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929\": rpc error: code = NotFound desc = could not find container \"1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929\": container with ID starting with 1484576c18fdffee9b4051db8f7f214ffcbd5a7db9626cb9908668220d6e6929 not found: ID does not exist"
Sep 15 12:22:20.206470 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.206448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkx9\" (UniqueName: \"kubernetes.io/projected/2039c1b8-aec2-45a6-b645-cd3af08be3e5-kube-api-access-pxkx9\") pod \"test-curl-7938-b3b8ad\" (UID: \"2039c1b8-aec2-45a6-b645-cd3af08be3e5\") " pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:20.215440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.215411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkx9\" (UniqueName: \"kubernetes.io/projected/2039c1b8-aec2-45a6-b645-cd3af08be3e5-kube-api-access-pxkx9\") pod \"test-curl-7938-b3b8ad\" (UID: \"2039c1b8-aec2-45a6-b645-cd3af08be3e5\") " pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:20.287264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.287231    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:20.575615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.575587    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7938-b3b8ad"]
Sep 15 12:22:20.576799 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:20.576772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2039c1b8_aec2_45a6_b645_cd3af08be3e5.slice/crio-53fb894b1ef3ba286c40a5bf00cccdc87ef82b23a6b911f186549f2c4b5f9c38 WatchSource:0}: Error finding container 53fb894b1ef3ba286c40a5bf00cccdc87ef82b23a6b911f186549f2c4b5f9c38: Status 404 returned error can't find the container with id 53fb894b1ef3ba286c40a5bf00cccdc87ef82b23a6b911f186549f2c4b5f9c38
Sep 15 12:22:20.945698 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:20.945600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:21.199015 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.198984    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-b3b8ad" event={"ID":"2039c1b8-aec2-45a6-b645-cd3af08be3e5","Type":"ContainerStarted","Data":"6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4"}
Sep 15 12:22:21.199129 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.199049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7938-b3b8ad" event={"ID":"2039c1b8-aec2-45a6-b645-cd3af08be3e5","Type":"ContainerStarted","Data":"53fb894b1ef3ba286c40a5bf00cccdc87ef82b23a6b911f186549f2c4b5f9c38"}
Sep 15 12:22:21.200175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.200144    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-fac56d" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" containerName="test-curl-7938-fac56d" containerID="cri-o://5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d" gracePeriod=2
Sep 15 12:22:21.227203 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:21.227174    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-fac56d\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:21.228623 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.228584    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7938-b3b8ad" podStartSLOduration=2.228568055 podStartE2EDuration="2.228568055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:21.225165105 +0000 UTC m=+1221.921724666" watchObservedRunningTime="2026-09-15 12:22:21.228568055 +0000 UTC m=+1221.925127633"
Sep 15 12:22:21.363767 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.363743    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:21.365826 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:21.365799    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-fac56d\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:21.520711 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.520631    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hbkq\" (UniqueName: \"kubernetes.io/projected/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe-kube-api-access-5hbkq\") pod \"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe\" (UID: \"d4b59a7e-53ce-42ff-81a0-b01c116b3ebe\") "
Sep 15 12:22:21.522790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.522760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe-kube-api-access-5hbkq" (OuterVolumeSpecName: "kube-api-access-5hbkq") pod "d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" (UID: "d4b59a7e-53ce-42ff-81a0-b01c116b3ebe"). InnerVolumeSpecName "kube-api-access-5hbkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:21.621950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.621907    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hbkq\" (UniqueName: \"kubernetes.io/projected/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe-kube-api-access-5hbkq\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:21.808182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.808095    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" path="/var/lib/kubelet/pods/d4b59a7e-53ce-42ff-81a0-b01c116b3ebe/volumes"
Sep 15 12:22:21.891400 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.891363    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7938-b3b8ad"]
Sep 15 12:22:21.896857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:21.896826    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7938-b3b8ad"]
Sep 15 12:22:22.092586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.092545    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:22.208059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.208010    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4b59a7e-53ce-42ff-81a0-b01c116b3ebe" containerID="a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707" exitCode=137
Sep 15 12:22:22.208264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.208084    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-074b38"
Sep 15 12:22:22.208264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.208093    2583 scope.go:117] "RemoveContainer" containerID="a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707"
Sep 15 12:22:22.208396 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:22.208376    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7938-b3b8ad" container="test-curl-7938-b3b8ad"
Sep 15 12:22:22.218882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.218861    2583 scope.go:117] "RemoveContainer" containerID="a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707"
Sep 15 12:22:22.219229 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:22.219202    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707\": container with ID starting with a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707 not found: ID does not exist" containerID="a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707"
Sep 15 12:22:22.219321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.219237    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707"} err="failed to get container status \"a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707\": rpc error: code = NotFound desc = could not find container \"a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707\": container with ID starting with a6436e9b58e9b1aed9fd29c7eeb2001d8ea9f144a5f6751eb2e7a7e12e5dc707 not found: ID does not exist"
Sep 15 12:22:22.500404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.500311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:22.838875 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.838842    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"]
Sep 15 12:22:22.839610 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.839588    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" containerName="test-curl-7938-b3b8ad"
Sep 15 12:22:22.839671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.839617    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" containerName="test-curl-7938-b3b8ad"
Sep 15 12:22:22.839783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.839770    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" containerName="test-curl-7938-b3b8ad"
Sep 15 12:22:22.844815 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.844792    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:22.847374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.847353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-80b72486-openshift-default-dockercfg-wtspm\""
Sep 15 12:22:22.877173 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.877132    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"]
Sep 15 12:22:22.897420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:22.897374    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:23.035825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.035782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e452a9-1608-470d-9515-49169b37bf50-istio-podinfo\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.035825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.035821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-data\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.035851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e452a9-1608-470d-9515-49169b37bf50-istiod-ca-cert\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.035968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-credential-socket\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.036013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-envoy\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.036067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-certs\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036125 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.036098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-socket\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.036126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czz2s\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-kube-api-access-czz2s\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.036423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.036163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-istio-token\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.136939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e452a9-1608-470d-9515-49169b37bf50-istio-podinfo\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.136989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-data\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e452a9-1608-470d-9515-49169b37bf50-istiod-ca-cert\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-credential-socket\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-envoy\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-certs\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-socket\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czz2s\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-kube-api-access-czz2s\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-istio-token\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.137628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.137488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-data\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.138179 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.138154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-certs\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.138254 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.138184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e452a9-1608-470d-9515-49169b37bf50-istiod-ca-cert\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.138813 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.138789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-socket\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.138944 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.138914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-credential-socket\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.140238 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.140217    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e452a9-1608-470d-9515-49169b37bf50-istio-podinfo\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.141166 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.141139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-envoy\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.162587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.162561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-istio-token\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.162696 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.162626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czz2s\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-kube-api-access-czz2s\") pod \"e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") " pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.216547 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.216512    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7938-b3b8ad" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" containerName="test-curl-7938-b3b8ad" containerID="cri-o://6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4" gracePeriod=2
Sep 15 12:22:23.222086 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:23.222057    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-b3b8ad\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:23.265390 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.265355    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:23.399919 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.399894    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:23.402359 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:23.402332    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7938-b3b8ad\" is forbidden: User \"system:node:ip-10-0-140-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-229.ec2.internal' and this object" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:23.440729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.440696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22r4c\" (UniqueName: \"kubernetes.io/projected/9cf39880-a473-4fc8-8b24-ef08ef18e99f-kube-api-access-22r4c\") pod \"9cf39880-a473-4fc8-8b24-ef08ef18e99f\" (UID: \"9cf39880-a473-4fc8-8b24-ef08ef18e99f\") "
Sep 15 12:22:23.442836 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.442816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf39880-a473-4fc8-8b24-ef08ef18e99f-kube-api-access-22r4c" (OuterVolumeSpecName: "kube-api-access-22r4c") pod "9cf39880-a473-4fc8-8b24-ef08ef18e99f" (UID: "9cf39880-a473-4fc8-8b24-ef08ef18e99f"). InnerVolumeSpecName "kube-api-access-22r4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:23.460065 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.460010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:23.542059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.542011    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22r4c\" (UniqueName: \"kubernetes.io/projected/9cf39880-a473-4fc8-8b24-ef08ef18e99f-kube-api-access-22r4c\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:23.748320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.748296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"]
Sep 15 12:22:23.750074 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:23.750047    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e452a9_1608_470d_9515_49169b37bf50.slice/crio-d7068fdbc0907e11d0e5edc870a40149aef9c0710f92cb0a592ac111a1f5b082 WatchSource:0}: Error finding container d7068fdbc0907e11d0e5edc870a40149aef9c0710f92cb0a592ac111a1f5b082: Status 404 returned error can't find the container with id d7068fdbc0907e11d0e5edc870a40149aef9c0710f92cb0a592ac111a1f5b082
Sep 15 12:22:23.759565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.753111    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:22:23.759565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.753269    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:22:23.759565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.753359    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:22:23.808235 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:23.808201    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" path="/var/lib/kubelet/pods/9cf39880-a473-4fc8-8b24-ef08ef18e99f/volumes"
Sep 15 12:22:24.223710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.223676    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cf39880-a473-4fc8-8b24-ef08ef18e99f" containerID="5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d" exitCode=137
Sep 15 12:22:24.224209 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.223730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-fac56d"
Sep 15 12:22:24.224209 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.223775    2583 scope.go:117] "RemoveContainer" containerID="5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d"
Sep 15 12:22:24.225690 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.225649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" event={"ID":"d1e452a9-1608-470d-9515-49169b37bf50","Type":"ContainerStarted","Data":"9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"}
Sep 15 12:22:24.225827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.225725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" event={"ID":"d1e452a9-1608-470d-9515-49169b37bf50","Type":"ContainerStarted","Data":"d7068fdbc0907e11d0e5edc870a40149aef9c0710f92cb0a592ac111a1f5b082"}
Sep 15 12:22:24.233212 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.233193    2583 scope.go:117] "RemoveContainer" containerID="5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d"
Sep 15 12:22:24.233493 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:24.233474    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d\": container with ID starting with 5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d not found: ID does not exist" containerID="5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d"
Sep 15 12:22:24.233553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.233506    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d"} err="failed to get container status \"5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d\": rpc error: code = NotFound desc = could not find container \"5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d\": container with ID starting with 5efba79b1097c68e0453382edcb1991e57d084699dd8e93c548c7b12c3f2f84d not found: ID does not exist"
Sep 15 12:22:24.257575 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.257527    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podStartSLOduration=2.257512953 podStartE2EDuration="2.257512953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:24.254946484 +0000 UTC m=+1224.951506104" watchObservedRunningTime="2026-09-15 12:22:24.257512953 +0000 UTC m=+1224.954072531"
Sep 15 12:22:24.460370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.460272    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:24.465708 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:24.465685    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:25.235249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.235224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:25.236074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.236056    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:22:25.292334 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.292291    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:25.373001 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.372970    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:25.423945 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.423921    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:25.459131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.459105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxkx9\" (UniqueName: \"kubernetes.io/projected/2039c1b8-aec2-45a6-b645-cd3af08be3e5-kube-api-access-pxkx9\") pod \"2039c1b8-aec2-45a6-b645-cd3af08be3e5\" (UID: \"2039c1b8-aec2-45a6-b645-cd3af08be3e5\") "
Sep 15 12:22:25.461146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.461082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2039c1b8-aec2-45a6-b645-cd3af08be3e5-kube-api-access-pxkx9" (OuterVolumeSpecName: "kube-api-access-pxkx9") pod "2039c1b8-aec2-45a6-b645-cd3af08be3e5" (UID: "2039c1b8-aec2-45a6-b645-cd3af08be3e5"). InnerVolumeSpecName "kube-api-access-pxkx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:25.560474 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.560391    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkx9\" (UniqueName: \"kubernetes.io/projected/2039c1b8-aec2-45a6-b645-cd3af08be3e5-kube-api-access-pxkx9\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:25.584245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.584196    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:25.685302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.685264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:25.807341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:25.807306    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" path="/var/lib/kubelet/pods/2039c1b8-aec2-45a6-b645-cd3af08be3e5/volumes"
Sep 15 12:22:26.242740 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.242650    2583 generic.go:356] "Generic (PLEG): container finished" podID="2039c1b8-aec2-45a6-b645-cd3af08be3e5" containerID="6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4" exitCode=137
Sep 15 12:22:26.242740 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.242703    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7938-b3b8ad"
Sep 15 12:22:26.243283 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.242707    2583 scope.go:117] "RemoveContainer" containerID="6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4"
Sep 15 12:22:26.251900 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.251875    2583 scope.go:117] "RemoveContainer" containerID="6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4"
Sep 15 12:22:26.252223 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:26.252199    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4\": container with ID starting with 6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4 not found: ID does not exist" containerID="6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4"
Sep 15 12:22:26.252299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.252231    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4"} err="failed to get container status \"6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4\": rpc error: code = NotFound desc = could not find container \"6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4\": container with ID starting with 6f1c409a4ec9b69f6efc12464713b9a300fdc0ae4531f19d31866b0e842f7df4 not found: ID does not exist"
Sep 15 12:22:26.313510 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.313474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:26.408628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.408589    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:26.822520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:26.822473    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:27.677482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:27.677451    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:28.544874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:28.544842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:29.574448 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:29.574411    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:30.512282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:30.512233    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:31.394708 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.394670    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"]
Sep 15 12:22:31.398582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.398556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.401791 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.401769    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-80b72486-serving-cert\""
Sep 15 12:22:31.416143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.416117    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"]
Sep 15 12:22:31.421736 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.421713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-metrics-certs\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.421850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.421746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzw9\" (UniqueName: \"kubernetes.io/projected/33bb28ec-daad-4861-ba47-b1f002afb333-kube-api-access-4xzw9\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.421904 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.421850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bb28ec-daad-4861-ba47-b1f002afb333-openshift-service-ca\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.421904 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.421894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.438342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.438266    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:31.522606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.522572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bb28ec-daad-4861-ba47-b1f002afb333-openshift-service-ca\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.522606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.522619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.522884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.522754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-metrics-certs\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.522884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.522782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzw9\" (UniqueName: \"kubernetes.io/projected/33bb28ec-daad-4861-ba47-b1f002afb333-kube-api-access-4xzw9\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.522884 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:31.522794    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-serving-cert: secret "maas-api-e2e-mt-80b72486-serving-cert" not found
Sep 15 12:22:31.522884 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:31.522868    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls podName:33bb28ec-daad-4861-ba47-b1f002afb333 nodeName:}" failed. No retries permitted until 2026-09-15 12:22:32.022849637 +0000 UTC m=+1232.719409215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls") pod "maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" (UID: "33bb28ec-daad-4861-ba47-b1f002afb333") : secret "maas-api-e2e-mt-80b72486-serving-cert" not found
Sep 15 12:22:31.523304 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.523281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bb28ec-daad-4861-ba47-b1f002afb333-openshift-service-ca\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.525519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.525489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-metrics-certs\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.533746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.533720    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzw9\" (UniqueName: \"kubernetes.io/projected/33bb28ec-daad-4861-ba47-b1f002afb333-kube-api-access-4xzw9\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:31.543821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.543799    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:31.973797 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:31.973761    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:32.029057 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:32.029010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:32.031404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:32.031379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls\") pod \"maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:32.067093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:32.067061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:32.311883 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:32.311762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:32.444085 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:32.444044    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:32.592123 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:32.592101    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"]
Sep 15 12:22:32.594414 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:32.594390    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bb28ec_daad_4861_ba47_b1f002afb333.slice/crio-91d3f98fa1c3b61b5ccf80860d98859c1660471a9276c89e5564a2628e6c4541 WatchSource:0}: Error finding container 91d3f98fa1c3b61b5ccf80860d98859c1660471a9276c89e5564a2628e6c4541: Status 404 returned error can't find the container with id 91d3f98fa1c3b61b5ccf80860d98859c1660471a9276c89e5564a2628e6c4541
Sep 15 12:22:33.287577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:33.287533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" event={"ID":"33bb28ec-daad-4861-ba47-b1f002afb333","Type":"ContainerStarted","Data":"c69205b931d4a170bab512bd2b8024f07382097a718873010a18e3bff9ef7076"}
Sep 15 12:22:33.287750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:33.287598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" event={"ID":"33bb28ec-daad-4861-ba47-b1f002afb333","Type":"ContainerStarted","Data":"91d3f98fa1c3b61b5ccf80860d98859c1660471a9276c89e5564a2628e6c4541"}
Sep 15 12:22:33.287750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:33.287639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:33.417079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:33.416980    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:34.322341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:34.322304    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:36.026952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.026919    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:36.688146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.688080    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" podStartSLOduration=5.197069408 podStartE2EDuration="5.688058472s" totalImagesPullingTime="490.989064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:33.317268358 +0000 UTC m=+1234.013827938" watchObservedRunningTime="2026-09-15 12:22:36.688058472 +0000 UTC m=+1237.384618052"
Sep 15 12:22:36.689650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.689624    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"]
Sep 15 12:22:36.689893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.689869    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" podUID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" containerName="maas-api" containerID="cri-o://0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d" gracePeriod=30
Sep 15 12:22:36.760598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.760565    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:36.972616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.972584    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:22:36.974279 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:36.974208    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:37.074514 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.074481    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:37.083287 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.083256    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-metrics-certs\") pod \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") "
Sep 15 12:22:37.083419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.083303    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-openshift-service-ca\") pod \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") "
Sep 15 12:22:37.083419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.083329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls\") pod \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") "
Sep 15 12:22:37.083419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.083403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrgpv\" (UniqueName: \"kubernetes.io/projected/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-kube-api-access-nrgpv\") pod \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\" (UID: \"aaa847bb-1f34-4b2e-8540-ec63c5c8e192\") "
Sep 15 12:22:37.083701 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.083673    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aaa847bb-1f34-4b2e-8540-ec63c5c8e192" (UID: "aaa847bb-1f34-4b2e-8540-ec63c5c8e192"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:37.085536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.085509    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aaa847bb-1f34-4b2e-8540-ec63c5c8e192" (UID: "aaa847bb-1f34-4b2e-8540-ec63c5c8e192"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:37.085633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.085548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aaa847bb-1f34-4b2e-8540-ec63c5c8e192" (UID: "aaa847bb-1f34-4b2e-8540-ec63c5c8e192"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:22:37.085696 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.085675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-kube-api-access-nrgpv" (OuterVolumeSpecName: "kube-api-access-nrgpv") pod "aaa847bb-1f34-4b2e-8540-ec63c5c8e192" (UID: "aaa847bb-1f34-4b2e-8540-ec63c5c8e192"). InnerVolumeSpecName "kube-api-access-nrgpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:37.179875 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.179813    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:37.186444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.185901    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:37.186444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.185937    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:37.186444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.185953    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:37.186444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.185967    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrgpv\" (UniqueName: \"kubernetes.io/projected/aaa847bb-1f34-4b2e-8540-ec63c5c8e192-kube-api-access-nrgpv\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:37.311201 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.311096    2583 generic.go:356] "Generic (PLEG): container finished" podID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" containerID="0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d" exitCode=0
Sep 15 12:22:37.311201 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.311154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"
Sep 15 12:22:37.311201 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.311189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" event={"ID":"aaa847bb-1f34-4b2e-8540-ec63c5c8e192","Type":"ContainerDied","Data":"0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d"}
Sep 15 12:22:37.311506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.311256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" event={"ID":"aaa847bb-1f34-4b2e-8540-ec63c5c8e192","Type":"ContainerDied","Data":"e18a4732fadd334e533b25cd0e90bc50e9395b8819dd207d57791259cb7863b2"}
Sep 15 12:22:37.311506 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.311286    2583 scope.go:117] "RemoveContainer" containerID="0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d"
Sep 15 12:22:37.321908 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.321886    2583 scope.go:117] "RemoveContainer" containerID="0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d"
Sep 15 12:22:37.322251 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:37.322228    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d\": container with ID starting with 0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d not found: ID does not exist" containerID="0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d"
Sep 15 12:22:37.322345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.322262    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d"} err="failed to get container status \"0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d\": rpc error: code = NotFound desc = could not find container \"0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d\": container with ID starting with 0eca575438ed3ccc2ca0b08e064c475046a158fdf53fb5edd811a97b70965d0d not found: ID does not exist"
Sep 15 12:22:37.340635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.340608    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"]
Sep 15 12:22:37.344175 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.344155    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt"]
Sep 15 12:22:37.455263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.455224    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:37.632343 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.632309    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:37.809306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:37.809268    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" path="/var/lib/kubelet/pods/aaa847bb-1f34-4b2e-8540-ec63c5c8e192/volumes"
Sep 15 12:22:39.299093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:39.299062    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:22:39.883266 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:39.883216    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69d7ba-c6c655d65-lnqzt" podUID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": context deadline exceeded"
Sep 15 12:22:40.414237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:40.414201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:40.514649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:40.514605    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:41.309279 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:41.309241    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:42.155764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:42.155726    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:44.628475 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:44.628440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:44.741473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:44.741426    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:44.863539 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:44.863507    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"]
Sep 15 12:22:44.863742 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:44.863719    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" containerID="cri-o://87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb" gracePeriod=30
Sep 15 12:22:44.887732 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:44.887655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:45.286080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.285971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:45.401998 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.401965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:45.724560 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.724527    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"]
Sep 15 12:22:45.725263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.725245    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" containerName="maas-api"
Sep 15 12:22:45.725263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.725265    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" containerName="maas-api"
Sep 15 12:22:45.725419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.725408    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa847bb-1f34-4b2e-8540-ec63c5c8e192" containerName="maas-api"
Sep 15 12:22:45.738634 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.738604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.746946 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.746920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1dc430-openshift-default-dockercfg-wvbr7\""
Sep 15 12:22:45.750076 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.750052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"]
Sep 15 12:22:45.801660 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.801620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:45.876840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.876807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-credential-socket\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.876840 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.876841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8dd5701-61f4-4956-911f-9af19db86494-istio-podinfo\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.876886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-envoy\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.876905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-istio-token\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.877019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kx26\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-kube-api-access-9kx26\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877214 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.877104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-data\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877214 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.877146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-socket\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877214 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.877179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8dd5701-61f4-4956-911f-9af19db86494-istiod-ca-cert\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.877328 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.877237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-certs\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.978612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-data\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.978612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-socket\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.978874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8dd5701-61f4-4956-911f-9af19db86494-istiod-ca-cert\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.978874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-certs\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.978874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-credential-socket\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.978874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8dd5701-61f4-4956-911f-9af19db86494-istio-podinfo\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-envoy\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-istio-token\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.978941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-socket\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.979018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kx26\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-kube-api-access-9kx26\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.979172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-credential-socket\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.979234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-data\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979547 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.979523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-certs\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.979682 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.979662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8dd5701-61f4-4956-911f-9af19db86494-istiod-ca-cert\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.981418 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.981398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-envoy\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.981557 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.981534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8dd5701-61f4-4956-911f-9af19db86494-istio-podinfo\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.991110 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.991083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-istio-token\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:45.991401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:45.991384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kx26\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-kube-api-access-9kx26\") pod \"e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") " pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:46.012042 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.011984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:46.050580 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.050556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:46.331600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.331569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"]
Sep 15 12:22:46.333317 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:22:46.333265    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dd5701_61f4_4956_911f_9af19db86494.slice/crio-2b66d13a64fef50f33f7bfe8c7e3c3cd54c2fa6d3ba8a8b664a53c1ae1ede68f WatchSource:0}: Error finding container 2b66d13a64fef50f33f7bfe8c7e3c3cd54c2fa6d3ba8a8b664a53c1ae1ede68f: Status 404 returned error can't find the container with id 2b66d13a64fef50f33f7bfe8c7e3c3cd54c2fa6d3ba8a8b664a53c1ae1ede68f
Sep 15 12:22:46.335848 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.335815    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:22:46.335935 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.335884    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:22:46.335935 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.335912    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:22:46.370918 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:46.370891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" event={"ID":"a8dd5701-61f4-4956-911f-9af19db86494","Type":"ContainerStarted","Data":"2b66d13a64fef50f33f7bfe8c7e3c3cd54c2fa6d3ba8a8b664a53c1ae1ede68f"}
Sep 15 12:22:47.379623 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:47.379587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" event={"ID":"a8dd5701-61f4-4956-911f-9af19db86494","Type":"ContainerStarted","Data":"19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"}
Sep 15 12:22:47.410654 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:47.410600    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podStartSLOduration=2.41058429 podStartE2EDuration="2.41058429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:47.409509671 +0000 UTC m=+1248.106069264" watchObservedRunningTime="2026-09-15 12:22:47.41058429 +0000 UTC m=+1248.107143925"
Sep 15 12:22:47.975564 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:47.975524    2583 patch_prober.go:28] interesting pod/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:22:47.975737 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:47.975584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:22:48.051492 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:48.051458    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:48.056918 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:48.056888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:48.386168 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:48.386131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:48.387256 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:48.387236    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:22:48.751874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:48.751766    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:49.380824 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:49.380784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:49.487275 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:49.487237    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:49.787596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:49.787471    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:50.135915 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.135890    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:22:50.225673 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-credential-socket\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.225854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-podinfo\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.225854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225764    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-data\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.225854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dde6444-40e8-433a-8f97-3759c48d9c23-istiod-ca-cert\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.225854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-envoy\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.225854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtt6m\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-kube-api-access-dtt6m\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.226156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-certs\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.226156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-token\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.226156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:50.226156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.225966    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-socket\") pod \"2dde6444-40e8-433a-8f97-3759c48d9c23\" (UID: \"2dde6444-40e8-433a-8f97-3759c48d9c23\") "
Sep 15 12:22:50.226156 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226119    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:50.226424 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226275    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:50.226424 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:50.226572 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dde6444-40e8-433a-8f97-3759c48d9c23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:22:50.226631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226544    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.226631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226606    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.226631 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226622    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.226788 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.226638    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.228223 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.228196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:22:50.228333 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.228250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:50.228333 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.228321    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:22:50.228408 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.228337    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-kube-api-access-dtt6m" (OuterVolumeSpecName: "kube-api-access-dtt6m") pod "2dde6444-40e8-433a-8f97-3759c48d9c23" (UID: "2dde6444-40e8-433a-8f97-3759c48d9c23"). InnerVolumeSpecName "kube-api-access-dtt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:22:50.327278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.327238    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.327278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.327276    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dde6444-40e8-433a-8f97-3759c48d9c23-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.327278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.327288    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.327525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.327297    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtt6m\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-kube-api-access-dtt6m\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.327525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.327319    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dde6444-40e8-433a-8f97-3759c48d9c23-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:22:50.400551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.400455    2583 generic.go:356] "Generic (PLEG): container finished" podID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerID="87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb" exitCode=0
Sep 15 12:22:50.400551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.400527    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"
Sep 15 12:22:50.400759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.400541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" event={"ID":"2dde6444-40e8-433a-8f97-3759c48d9c23","Type":"ContainerDied","Data":"87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb"}
Sep 15 12:22:50.400759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.400601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd" event={"ID":"2dde6444-40e8-433a-8f97-3759c48d9c23","Type":"ContainerDied","Data":"8b3e76861ecbd4c73820c2e3963fb5b55a344229493c02fc6a7de360d6502dd9"}
Sep 15 12:22:50.400759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.400628    2583 scope.go:117] "RemoveContainer" containerID="87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb"
Sep 15 12:22:50.411237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.411218    2583 scope.go:117] "RemoveContainer" containerID="cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"
Sep 15 12:22:50.423888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.423869    2583 scope.go:117] "RemoveContainer" containerID="87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb"
Sep 15 12:22:50.424282 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:50.424254    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb\": container with ID starting with 87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb not found: ID does not exist" containerID="87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb"
Sep 15 12:22:50.424397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.424293    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb"} err="failed to get container status \"87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb\": rpc error: code = NotFound desc = could not find container \"87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb\": container with ID starting with 87e7d50e54fa4d3cc893e17b605497242d74cffc904730dbfbf9f5cd0cfeabfb not found: ID does not exist"
Sep 15 12:22:50.424397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.424318    2583 scope.go:117] "RemoveContainer" containerID="cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"
Sep 15 12:22:50.424704 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:50.424680    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3\": container with ID starting with cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3 not found: ID does not exist" containerID="cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"
Sep 15 12:22:50.424793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.424746    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3"} err="failed to get container status \"cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3\": rpc error: code = NotFound desc = could not find container \"cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3\": container with ID starting with cf80866dbbd7b960f0a90e4d35a0f4c64fefe5120e8755428a0f6714434a6ec3 not found: ID does not exist"
Sep 15 12:22:50.427154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.427133    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"]
Sep 15 12:22:50.434678 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.434654    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-69d7ba-openshift-default-544b756846-g5vsd"]
Sep 15 12:22:50.626177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.626141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:50.847172 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:50.847142    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:51.116143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:51.116039    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:51.800598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:51.800555    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:51.808328 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:51.808300    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" path="/var/lib/kubelet/pods/2dde6444-40e8-433a-8f97-3759c48d9c23/volumes"
Sep 15 12:22:52.895638 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:52.895601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:53.904379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:53.904341    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:54.993010 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.992972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"]
Sep 15 12:22:54.993727 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.993704    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy"
Sep 15 12:22:54.993832 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.993730    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy"
Sep 15 12:22:54.993832 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.993743    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy"
Sep 15 12:22:54.993832 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.993752    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy"
Sep 15 12:22:54.993984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.993908    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy"
Sep 15 12:22:54.999109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:54.999080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.002449 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.002426    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1dc430-serving-cert\""
Sep 15 12:22:55.021406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.020036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"]
Sep 15 12:22:55.050859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.050820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:55.079771 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.079741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448aa45-786d-474e-a9eb-19db08677274-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.079951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.079783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.079951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.079907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.079951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.079945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wmw\" (UniqueName: \"kubernetes.io/projected/d448aa45-786d-474e-a9eb-19db08677274-kube-api-access-r6wmw\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.144889 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.144846    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:55.180915 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.180878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.181132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.180941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wmw\" (UniqueName: \"kubernetes.io/projected/d448aa45-786d-474e-a9eb-19db08677274-kube-api-access-r6wmw\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.181132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.181009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448aa45-786d-474e-a9eb-19db08677274-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.181132 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.181078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.181307 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:55.181268    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-serving-cert: secret "maas-api-e2e-shared-a-w3-1dc430-serving-cert" not found
Sep 15 12:22:55.181367 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:55.181360    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls podName:d448aa45-786d-474e-a9eb-19db08677274 nodeName:}" failed. No retries permitted until 2026-09-15 12:22:55.681337711 +0000 UTC m=+1256.377897268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" (UID: "d448aa45-786d-474e-a9eb-19db08677274") : secret "maas-api-e2e-shared-a-w3-1dc430-serving-cert" not found
Sep 15 12:22:55.181800 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.181774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448aa45-786d-474e-a9eb-19db08677274-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.183419 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.183385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.193817 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.193788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wmw\" (UniqueName: \"kubernetes.io/projected/d448aa45-786d-474e-a9eb-19db08677274-kube-api-access-r6wmw\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.659795 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.659762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:55.686035 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.685991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:55.686220 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:55.686150    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-serving-cert: secret "maas-api-e2e-shared-a-w3-1dc430-serving-cert" not found
Sep 15 12:22:55.686291 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:22:55.686227    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls podName:d448aa45-786d-474e-a9eb-19db08677274 nodeName:}" failed. No retries permitted until 2026-09-15 12:22:56.686205561 +0000 UTC m=+1257.382765120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" (UID: "d448aa45-786d-474e-a9eb-19db08677274") : secret "maas-api-e2e-shared-a-w3-1dc430-serving-cert" not found
Sep 15 12:22:55.762254 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:55.762214    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:56.130048 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:56.129577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:56.243249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:56.243215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:56.694061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:56.693997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:56.696472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:56.696443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:56.821726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:56.821682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:57.114040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:57.105627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"]
Sep 15 12:22:57.146189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:57.146152    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:57.451396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:57.451309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" event={"ID":"d448aa45-786d-474e-a9eb-19db08677274","Type":"ContainerStarted","Data":"6864bab9421a6dc2d424ecfadd098c406a21cbabc12b31f388d94e0398c32dd0"}
Sep 15 12:22:58.298968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:58.298929    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:22:58.459174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:58.459135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" event={"ID":"d448aa45-786d-474e-a9eb-19db08677274","Type":"ContainerStarted","Data":"da38958427e0c79d033ac10f50384086533ac615f7a341fe62d077bddbd8d477"}
Sep 15 12:22:58.459355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:58.459333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:22:59.305094 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:22:59.304985    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:02.491733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.491635    2583 generic.go:356] "Generic (PLEG): container finished" podID="d1e452a9-1608-470d-9515-49169b37bf50" containerID="9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74" exitCode=0
Sep 15 12:23:02.491733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.491705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" event={"ID":"d1e452a9-1608-470d-9515-49169b37bf50","Type":"ContainerDied","Data":"9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"}
Sep 15 12:23:02.492235 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.492216    2583 scope.go:117] "RemoveContainer" containerID="9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"
Sep 15 12:23:02.493392 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.493363    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:02.493459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.493433    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:02.493503 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.493463    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:02.534334 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:02.533765    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" podStartSLOduration=8.081368913 podStartE2EDuration="8.53374182s" totalImagesPullingTime="452.372907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:58.512694766 +0000 UTC m=+1259.209254347" watchObservedRunningTime="2026-09-15 12:23:02.53374182 +0000 UTC m=+1263.230301398"
Sep 15 12:23:03.500406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:03.500368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" event={"ID":"d1e452a9-1608-470d-9515-49169b37bf50","Type":"ContainerStarted","Data":"0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a"}
Sep 15 12:23:04.460805 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:04.460768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:23:04.461051 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:04.460966    2583 patch_prober.go:28] interesting pod/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 12:23:04.461109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:04.461017    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 12:23:04.479080 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:04.479054    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:23:05.461173 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:05.461137    2583 patch_prober.go:28] interesting pod/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 12:23:05.461574 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:05.461189    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 12:23:06.460956 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:06.460918    2583 patch_prober.go:28] interesting pod/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 12:23:06.461186 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:06.460973    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 12:23:07.460973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:07.460936    2583 patch_prober.go:28] interesting pod/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 12:23:07.461396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:07.460993    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 12:23:08.460761 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:08.460720    2583 patch_prober.go:28] interesting pod/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 12:23:08.460947 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:08.460791    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 12:23:09.465189 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:09.465155    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:23:09.465569 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:09.465460    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:23:09.466095 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:09.466077    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:23:15.435233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:15.435191    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:18.068232 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:18.068192    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:18.790405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:18.790363    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:19.132687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:19.132654    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:19.384803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:19.384709    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:19.735730 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:19.735633    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:20.000294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.000196    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"]
Sep 15 12:23:20.000927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.000904    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dde6444-40e8-433a-8f97-3759c48d9c23" containerName="istio-proxy"
Sep 15 12:23:20.003657 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.003635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.009670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.009643    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-969130-openshift-default-dockercfg-sdzkx\""
Sep 15 12:23:20.028382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.028352    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"]
Sep 15 12:23:20.061886 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.061847    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-podinfo\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f141f617-e485-4035-b5e5-722c8e1d48b1-istiod-ca-cert\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151425    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-data\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqt54\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-kube-api-access-lqt54\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-credential-socket\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-certs\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-socket\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-token\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.154079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.151704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-envoy\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.228722 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.228683    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:20.252960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.252866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-data\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.252960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.252909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqt54\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-kube-api-access-lqt54\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.252960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.252943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-credential-socket\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.252975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-certs\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-socket\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-token\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-envoy\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-podinfo\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f141f617-e485-4035-b5e5-722c8e1d48b1-istiod-ca-cert\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-certs\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253912 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f141f617-e485-4035-b5e5-722c8e1d48b1-istiod-ca-cert\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.253989 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.253723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-credential-socket\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.254200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.254179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-socket\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.254200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.254192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-data\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.256034 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.256001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-podinfo\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.256198 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.256180    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-envoy\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.263439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.263414    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-token\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.263612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.263596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqt54\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-kube-api-access-lqt54\") pod \"e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") " pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.330396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.330357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:20.630244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.629552    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"]
Sep 15 12:23:20.639264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.639211    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:20.639387 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.639307    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:20.639387 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.639351    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:20.733193 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:20.733154    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:21.546282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:21.546247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:21.622703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:21.622660    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" event={"ID":"f141f617-e485-4035-b5e5-722c8e1d48b1","Type":"ContainerStarted","Data":"1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"}
Sep 15 12:23:21.622881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:21.622726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" event={"ID":"f141f617-e485-4035-b5e5-722c8e1d48b1","Type":"ContainerStarted","Data":"4194c5d0ef85c0160eaa90d59b7e8055f6416b000aa63c308c1831bc5cd7ae28"}
Sep 15 12:23:21.652439 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:21.652385    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podStartSLOduration=2.652366291 podStartE2EDuration="2.652366291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:23:21.650085813 +0000 UTC m=+1282.346645404" watchObservedRunningTime="2026-09-15 12:23:21.652366291 +0000 UTC m=+1282.348925868"
Sep 15 12:23:22.331275 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:22.331233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:22.337322 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:22.337283    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:22.513950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:22.513854    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:22.629615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:22.629517    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:22.630833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:22.630812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:23:23.443915 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:23.443879    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:23.706370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:23.706286    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:23.835341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:23.835307    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:24.645567 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:24.645507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:25.618485 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.618452    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"]
Sep 15 12:23:25.621815 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.621788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.624699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.624677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-969130-serving-cert\""
Sep 15 12:23:25.637074 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.637049    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"]
Sep 15 12:23:25.658718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.658669    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:25.662089 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.662057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" event={"ID":"a8dd5701-61f4-4956-911f-9af19db86494","Type":"ContainerDied","Data":"19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"}
Sep 15 12:23:25.662241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.662088    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8dd5701-61f4-4956-911f-9af19db86494" containerID="19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a" exitCode=0
Sep 15 12:23:25.662548 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.662531    2583 scope.go:117] "RemoveContainer" containerID="19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"
Sep 15 12:23:25.664111 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.664073    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:25.664216 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.664170    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:25.664296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.664214    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:23:25.759653 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.759608    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:25.817677 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.817634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ab737a-9996-4d0c-bf36-bb397f96b45e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.817880 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.817684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.817880 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.817746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qk9\" (UniqueName: \"kubernetes.io/projected/76ab737a-9996-4d0c-bf36-bb397f96b45e-kube-api-access-94qk9\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.818010 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.817934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.919596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.919483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ab737a-9996-4d0c-bf36-bb397f96b45e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.919596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.919531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.919596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.919561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qk9\" (UniqueName: \"kubernetes.io/projected/76ab737a-9996-4d0c-bf36-bb397f96b45e-kube-api-access-94qk9\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.919904 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.919630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.920157 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:23:25.920130    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-serving-cert: secret "maas-api-e2e-shared-b-w3-969130-serving-cert" not found
Sep 15 12:23:25.920304 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:23:25.920244    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls podName:76ab737a-9996-4d0c-bf36-bb397f96b45e nodeName:}" failed. No retries permitted until 2026-09-15 12:23:26.420224546 +0000 UTC m=+1287.116784107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" (UID: "76ab737a-9996-4d0c-bf36-bb397f96b45e") : secret "maas-api-e2e-shared-b-w3-969130-serving-cert" not found
Sep 15 12:23:25.920398 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.920309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ab737a-9996-4d0c-bf36-bb397f96b45e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.922630 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.922606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:25.931394 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:25.931351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qk9\" (UniqueName: \"kubernetes.io/projected/76ab737a-9996-4d0c-bf36-bb397f96b45e-kube-api-access-94qk9\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:26.273810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:26.273719    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:26.375729 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:26.375691    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:26.423535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:26.423490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:26.423721 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:23:26.423649    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-serving-cert: secret "maas-api-e2e-shared-b-w3-969130-serving-cert" not found
Sep 15 12:23:26.423784 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:23:26.423728    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls podName:76ab737a-9996-4d0c-bf36-bb397f96b45e nodeName:}" failed. No retries permitted until 2026-09-15 12:23:27.423707345 +0000 UTC m=+1288.120266905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" (UID: "76ab737a-9996-4d0c-bf36-bb397f96b45e") : secret "maas-api-e2e-shared-b-w3-969130-serving-cert" not found
Sep 15 12:23:26.670685 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:26.670651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" event={"ID":"a8dd5701-61f4-4956-911f-9af19db86494","Type":"ContainerStarted","Data":"d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5"}
Sep 15 12:23:26.953483 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:26.953392    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:27.051523 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.051491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:23:27.051726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.051705    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 15 12:23:27.051770 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.051751    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 15 12:23:27.126796 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.126762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:27.433801 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.433752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:27.436540 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.436515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:27.735577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.735473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:27.958968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:27.958926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:28.021289 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.020663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"]
Sep 15 12:23:28.051306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.051272    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 15 12:23:28.051473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.051345    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 15 12:23:28.684418 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.684383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" event={"ID":"76ab737a-9996-4d0c-bf36-bb397f96b45e","Type":"ContainerStarted","Data":"a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643"}
Sep 15 12:23:28.684630 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.684442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" event={"ID":"76ab737a-9996-4d0c-bf36-bb397f96b45e","Type":"ContainerStarted","Data":"7c19612c94b75b7b4ac60710847fc2dfcea1c3fa10086084d97341e02015492f"}
Sep 15 12:23:28.684630 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.684592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:28.949929 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:28.949903    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:29.051167 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:29.051127    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 15 12:23:29.051333 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:29.051187    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 15 12:23:30.011646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:30.011599    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:30.051223 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:30.051186    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 15 12:23:30.051402 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:30.051242    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 15 12:23:30.114848 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:30.114812    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:31.033447 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:31.033395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:31.051721 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:31.051683    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 15 12:23:31.051858 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:31.051751    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 15 12:23:32.051207 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:32.051174    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 15 12:23:32.051635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:32.051609    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 15 12:23:32.071441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:32.071403    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:33.055579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:33.055547    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:23:33.056090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:33.055925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:23:33.056666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:33.056648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:23:33.088941 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:33.088889    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" podStartSLOduration=7.623153736 podStartE2EDuration="8.088875766s" totalImagesPullingTime="465.72203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:23:28.716409902 +0000 UTC m=+1289.412969519" watchObservedRunningTime="2026-09-15 12:23:33.088875766 +0000 UTC m=+1293.785435387"
Sep 15 12:23:34.695910 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:34.695882    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:23:46.484293 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:46.484258    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"]
Sep 15 12:23:46.509724 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:46.509688    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:48.151290 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:48.151252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:48.540430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:48.540346    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:49.229413 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:49.229379    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:50.205530 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:50.205491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:51.037625 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:51.037587    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:52.232256 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:52.232223    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:53.221639 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:53.221598    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:54.237755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:54.237693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:23:55.553009 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:23:55.552971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:01.503646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.503534    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr" podUID="5777ecd8-a48b-4a45-9dc8-5c75aad99367" containerName="main" containerID="cri-o://7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987" gracePeriod=45
Sep 15 12:24:01.746106 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.746078    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:24:01.799402 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799311    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-dshm\") pod \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") "
Sep 15 12:24:01.799402 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799356    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tls-certs\") pod \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") "
Sep 15 12:24:01.799402 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799385    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-home\") pod \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") "
Sep 15 12:24:01.799671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tmp-dir\") pod \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") "
Sep 15 12:24:01.799671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-model-cache\") pod \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") "
Sep 15 12:24:01.799671 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799616    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t6s2\" (UniqueName: \"kubernetes.io/projected/5777ecd8-a48b-4a45-9dc8-5c75aad99367-kube-api-access-5t6s2\") pod \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\" (UID: \"5777ecd8-a48b-4a45-9dc8-5c75aad99367\") "
Sep 15 12:24:01.799827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799702    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-home" (OuterVolumeSpecName: "home") pod "5777ecd8-a48b-4a45-9dc8-5c75aad99367" (UID: "5777ecd8-a48b-4a45-9dc8-5c75aad99367"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:01.799827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5777ecd8-a48b-4a45-9dc8-5c75aad99367" (UID: "5777ecd8-a48b-4a45-9dc8-5c75aad99367"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:01.799940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.799891    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-model-cache" (OuterVolumeSpecName: "model-cache") pod "5777ecd8-a48b-4a45-9dc8-5c75aad99367" (UID: "5777ecd8-a48b-4a45-9dc8-5c75aad99367"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:01.800154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.800106    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-home\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:01.800271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.800154    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tmp-dir\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:01.800271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.800171    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-model-cache\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:01.802010 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.801981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5777ecd8-a48b-4a45-9dc8-5c75aad99367-kube-api-access-5t6s2" (OuterVolumeSpecName: "kube-api-access-5t6s2") pod "5777ecd8-a48b-4a45-9dc8-5c75aad99367" (UID: "5777ecd8-a48b-4a45-9dc8-5c75aad99367"). InnerVolumeSpecName "kube-api-access-5t6s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:24:01.802147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.802048    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5777ecd8-a48b-4a45-9dc8-5c75aad99367" (UID: "5777ecd8-a48b-4a45-9dc8-5c75aad99367"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:24:01.802201 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.802150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-dshm" (OuterVolumeSpecName: "dshm") pod "5777ecd8-a48b-4a45-9dc8-5c75aad99367" (UID: "5777ecd8-a48b-4a45-9dc8-5c75aad99367"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:24:01.889349 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.889315    2583 generic.go:356] "Generic (PLEG): container finished" podID="5777ecd8-a48b-4a45-9dc8-5c75aad99367" containerID="7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987" exitCode=0
Sep 15 12:24:01.889538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.889364    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"
Sep 15 12:24:01.889538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.889374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr" event={"ID":"5777ecd8-a48b-4a45-9dc8-5c75aad99367","Type":"ContainerDied","Data":"7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987"}
Sep 15 12:24:01.889538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.889415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr" event={"ID":"5777ecd8-a48b-4a45-9dc8-5c75aad99367","Type":"ContainerDied","Data":"901dda685312f8ab0992ec063061f1c882246f253fbc1d27d61e8771aebde485"}
Sep 15 12:24:01.889538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.889433    2583 scope.go:117] "RemoveContainer" containerID="7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987"
Sep 15 12:24:01.891749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.891627    2583 generic.go:356] "Generic (PLEG): container finished" podID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerID="1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d" exitCode=0
Sep 15 12:24:01.891749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.891704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" event={"ID":"f141f617-e485-4035-b5e5-722c8e1d48b1","Type":"ContainerDied","Data":"1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"}
Sep 15 12:24:01.892139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.892125    2583 scope.go:117] "RemoveContainer" containerID="1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"
Sep 15 12:24:01.893416 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.893387    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:24:01.893515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.893460    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:24:01.893515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.893493    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:24:01.899589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.899566    2583 scope.go:117] "RemoveContainer" containerID="7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987"
Sep 15 12:24:01.899897 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:24:01.899879    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987\": container with ID starting with 7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987 not found: ID does not exist" containerID="7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987"
Sep 15 12:24:01.899943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.899905    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987"} err="failed to get container status \"7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987\": rpc error: code = NotFound desc = could not find container \"7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987\": container with ID starting with 7375f39c3dca1e7d859c463c8cb12c587cf0e7a90dbffcbacb02b0e4cad22987 not found: ID does not exist"
Sep 15 12:24:01.901872 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.901845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t6s2\" (UniqueName: \"kubernetes.io/projected/5777ecd8-a48b-4a45-9dc8-5c75aad99367-kube-api-access-5t6s2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:01.901975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.901877    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5777ecd8-a48b-4a45-9dc8-5c75aad99367-dshm\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:01.901975 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.901892    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5777ecd8-a48b-4a45-9dc8-5c75aad99367-tls-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:01.937264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.937219    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"]
Sep 15 12:24:01.941015 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:01.940986    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9ba48f/e2e-conflict-a-a7f06f2e-kserve-7d448b47b4-tfgpr"]
Sep 15 12:24:02.899783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:02.899749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" event={"ID":"f141f617-e485-4035-b5e5-722c8e1d48b1","Type":"ContainerStarted","Data":"6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399"}
Sep 15 12:24:03.331949 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:03.331914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:24:03.332970 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:03.332946    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 15 12:24:03.333143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:03.333104    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 15 12:24:03.811453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:03.811354    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5777ecd8-a48b-4a45-9dc8-5c75aad99367" path="/var/lib/kubelet/pods/5777ecd8-a48b-4a45-9dc8-5c75aad99367/volumes"
Sep 15 12:24:04.331648 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:04.331613    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 15 12:24:04.332244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:04.332215    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 15 12:24:05.331107 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:05.331068    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 15 12:24:05.331450 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:05.331418    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 15 12:24:06.331825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:06.331686    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 15 12:24:06.331825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:06.331762    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 15 12:24:07.331599 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:07.331556    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 15 12:24:07.331822 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:07.331626    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 15 12:24:07.630133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:07.630012    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:24:08.332112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:08.332062    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 15 12:24:08.332345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:08.332156    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 15 12:24:09.336081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:09.336051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:24:09.337200 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:09.337168    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:24:12.537898 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:12.537858    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:13.364263 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:13.364222    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:14.265162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:14.265129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:14.361871 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:14.361834    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:15.150518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:15.150489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:16.293229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:16.293193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:17.140086 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:17.140050    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:18.059752 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:18.059711    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:19.032182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:19.032145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:19.852444 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:19.852410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:21.872178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:21.872133    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:27.859196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:27.859162    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"]
Sep 15 12:24:27.859639 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:27.859364    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" podUID="33bb28ec-daad-4861-ba47-b1f002afb333" containerName="maas-api" containerID="cri-o://c69205b931d4a170bab512bd2b8024f07382097a718873010a18e3bff9ef7076" gracePeriod=30
Sep 15 12:24:27.898438 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:27.898400    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:27.991516 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:27.991482    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:28.059857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.059821    2583 generic.go:356] "Generic (PLEG): container finished" podID="33bb28ec-daad-4861-ba47-b1f002afb333" containerID="c69205b931d4a170bab512bd2b8024f07382097a718873010a18e3bff9ef7076" exitCode=0
Sep 15 12:24:28.060202 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.060051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" event={"ID":"33bb28ec-daad-4861-ba47-b1f002afb333","Type":"ContainerDied","Data":"c69205b931d4a170bab512bd2b8024f07382097a718873010a18e3bff9ef7076"}
Sep 15 12:24:28.066050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.065984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:28.122844 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.122748    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:24:28.176650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.176612    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:28.258318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.258282    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:28.279669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.279544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls\") pod \"33bb28ec-daad-4861-ba47-b1f002afb333\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") "
Sep 15 12:24:28.279669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.279603    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-metrics-certs\") pod \"33bb28ec-daad-4861-ba47-b1f002afb333\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") "
Sep 15 12:24:28.279938 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.279737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xzw9\" (UniqueName: \"kubernetes.io/projected/33bb28ec-daad-4861-ba47-b1f002afb333-kube-api-access-4xzw9\") pod \"33bb28ec-daad-4861-ba47-b1f002afb333\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") "
Sep 15 12:24:28.279938 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.279822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bb28ec-daad-4861-ba47-b1f002afb333-openshift-service-ca\") pod \"33bb28ec-daad-4861-ba47-b1f002afb333\" (UID: \"33bb28ec-daad-4861-ba47-b1f002afb333\") "
Sep 15 12:24:28.280518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.280469    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bb28ec-daad-4861-ba47-b1f002afb333-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33bb28ec-daad-4861-ba47-b1f002afb333" (UID: "33bb28ec-daad-4861-ba47-b1f002afb333"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:24:28.282282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.282254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33bb28ec-daad-4861-ba47-b1f002afb333" (UID: "33bb28ec-daad-4861-ba47-b1f002afb333"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:24:28.282872 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.282846    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bb28ec-daad-4861-ba47-b1f002afb333-kube-api-access-4xzw9" (OuterVolumeSpecName: "kube-api-access-4xzw9") pod "33bb28ec-daad-4861-ba47-b1f002afb333" (UID: "33bb28ec-daad-4861-ba47-b1f002afb333"). InnerVolumeSpecName "kube-api-access-4xzw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:24:28.282956 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.282877    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33bb28ec-daad-4861-ba47-b1f002afb333" (UID: "33bb28ec-daad-4861-ba47-b1f002afb333"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:24:28.381759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.381665    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xzw9\" (UniqueName: \"kubernetes.io/projected/33bb28ec-daad-4861-ba47-b1f002afb333-kube-api-access-4xzw9\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:28.381759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.381697    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bb28ec-daad-4861-ba47-b1f002afb333-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:28.381759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.381711    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:28.381759 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.381720    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bb28ec-daad-4861-ba47-b1f002afb333-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:24:28.532172 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:28.532140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:29.067423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.067392    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"
Sep 15 12:24:29.067876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.067424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc" event={"ID":"33bb28ec-daad-4861-ba47-b1f002afb333","Type":"ContainerDied","Data":"91d3f98fa1c3b61b5ccf80860d98859c1660471a9276c89e5564a2628e6c4541"}
Sep 15 12:24:29.067876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.067484    2583 scope.go:117] "RemoveContainer" containerID="c69205b931d4a170bab512bd2b8024f07382097a718873010a18e3bff9ef7076"
Sep 15 12:24:29.099278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.099248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"]
Sep 15 12:24:29.103389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.103367    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-80b72486-86c4d69cc6-bcbbc"]
Sep 15 12:24:29.637566 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.637526    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:29.808445 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:29.808414    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bb28ec-daad-4861-ba47-b1f002afb333" path="/var/lib/kubelet/pods/33bb28ec-daad-4861-ba47-b1f002afb333/volumes"
Sep 15 12:24:30.639346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:30.639308    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:31.275127 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:31.275092    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:31.468070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:31.468008    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:32.406726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:32.406688    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:33.304488 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:33.304450    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:34.104897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:34.104860    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:36.335819 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:36.335785    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:24:57.558301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:24:57.558262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:13.102607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:13.102569    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:14.046921 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:14.046886    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:15.114090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:15.114054    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:20.128354 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:20.128313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:20.240240 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:20.240200    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:20.396123 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:20.396015    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"]
Sep 15 12:25:20.396327 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:20.396281    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" containerID="cri-o://0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a" gracePeriod=30
Sep 15 12:25:20.443660 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:20.443615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:21.704453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:21.704418    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:21.836789 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:21.836735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:22.549139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:22.549104    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:22.703749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:22.703706    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:22.886227 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:22.886187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:24.094578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:24.094542    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:24.468723 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:24.468631    2583 patch_prober.go:28] interesting pod/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:25:24.468894 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:24.468709    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:25:25.663543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.663483    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:25:25.775531 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-socket\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.775746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775546    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-istio-token\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.775746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775573    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czz2s\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-kube-api-access-czz2s\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.775746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775642    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e452a9-1608-470d-9515-49169b37bf50-istiod-ca-cert\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.775746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775678    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-certs\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.775746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-data\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.775746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775730    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-envoy\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.776130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:25:25.776130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775811    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-credential-socket\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.776130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775848    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e452a9-1608-470d-9515-49169b37bf50-istio-podinfo\") pod \"d1e452a9-1608-470d-9515-49169b37bf50\" (UID: \"d1e452a9-1608-470d-9515-49169b37bf50\") "
Sep 15 12:25:25.776130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.775990    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:25:25.776356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.776185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:25:25.776356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.776207    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e452a9-1608-470d-9515-49169b37bf50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:25:25.776356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.776253    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.776356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.776274    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.776356 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.776287    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:25:25.777968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.777944    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:25:25.778096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.777988    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-kube-api-access-czz2s" (OuterVolumeSpecName: "kube-api-access-czz2s") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "kube-api-access-czz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:25:25.778270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.778253    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:25:25.778331 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.778301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e452a9-1608-470d-9515-49169b37bf50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1e452a9-1608-470d-9515-49169b37bf50" (UID: "d1e452a9-1608-470d-9515-49169b37bf50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:25:25.877660 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877622    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.877660 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877661    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czz2s\" (UniqueName: \"kubernetes.io/projected/d1e452a9-1608-470d-9515-49169b37bf50-kube-api-access-czz2s\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.877897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877675    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e452a9-1608-470d-9515-49169b37bf50-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.877897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877688    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.877897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877704    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.877897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877716    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e452a9-1608-470d-9515-49169b37bf50-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:25.877897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:25.877729    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e452a9-1608-470d-9515-49169b37bf50-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:25:26.413877 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.413842    2583 generic.go:356] "Generic (PLEG): container finished" podID="d1e452a9-1608-470d-9515-49169b37bf50" containerID="0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a" exitCode=0
Sep 15 12:25:26.414090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.413888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" event={"ID":"d1e452a9-1608-470d-9515-49169b37bf50","Type":"ContainerDied","Data":"0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a"}
Sep 15 12:25:26.414090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.413906    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"
Sep 15 12:25:26.414090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.413930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2" event={"ID":"d1e452a9-1608-470d-9515-49169b37bf50","Type":"ContainerDied","Data":"d7068fdbc0907e11d0e5edc870a40149aef9c0710f92cb0a592ac111a1f5b082"}
Sep 15 12:25:26.414090 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.413951    2583 scope.go:117] "RemoveContainer" containerID="0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a"
Sep 15 12:25:26.423282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.423262    2583 scope.go:117] "RemoveContainer" containerID="9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"
Sep 15 12:25:26.431636 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.431613    2583 scope.go:117] "RemoveContainer" containerID="0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a"
Sep 15 12:25:26.432045 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:25:26.431997    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a\": container with ID starting with 0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a not found: ID does not exist" containerID="0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a"
Sep 15 12:25:26.432152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.432054    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a"} err="failed to get container status \"0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a\": rpc error: code = NotFound desc = could not find container \"0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a\": container with ID starting with 0f4a228cfa6d039cca852878a8cb2d23d1fc6e11674fa4600939a4f1c932aa5a not found: ID does not exist"
Sep 15 12:25:26.432152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.432080    2583 scope.go:117] "RemoveContainer" containerID="9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"
Sep 15 12:25:26.432401 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:25:26.432369    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74\": container with ID starting with 9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74 not found: ID does not exist" containerID="9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"
Sep 15 12:25:26.432489 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.432403    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74"} err="failed to get container status \"9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74\": rpc error: code = NotFound desc = could not find container \"9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74\": container with ID starting with 9df6105eeac42926bcf2562e0fbaffe20b8bbf4ab080e3b5f6a4b304ae3e6c74 not found: ID does not exist"
Sep 15 12:25:26.434924 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.434894    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"]
Sep 15 12:25:26.440618 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:26.440597    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-80b72486-openshift-default-5f9d6d988d-k6fz2"]
Sep 15 12:25:27.808213 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:27.808174    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e452a9-1608-470d-9515-49169b37bf50" path="/var/lib/kubelet/pods/d1e452a9-1608-470d-9515-49169b37bf50/volumes"
Sep 15 12:25:30.259517 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:30.259466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:33.597666 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:33.597632    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:33.697508 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:33.697474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:37.804333 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:37.804297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:37.918032 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:37.917999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:38.042362 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:38.042331    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:42.040532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.040500    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"]
Sep 15 12:25:42.041006 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.040992    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bb28ec-daad-4861-ba47-b1f002afb333" containerName="maas-api"
Sep 15 12:25:42.041006 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041007    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bb28ec-daad-4861-ba47-b1f002afb333" containerName="maas-api"
Sep 15 12:25:42.041130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041033    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5777ecd8-a48b-4a45-9dc8-5c75aad99367" containerName="main"
Sep 15 12:25:42.041130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041041    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5777ecd8-a48b-4a45-9dc8-5c75aad99367" containerName="main"
Sep 15 12:25:42.041130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041058    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy"
Sep 15 12:25:42.041130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041067    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy"
Sep 15 12:25:42.041130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041076    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy"
Sep 15 12:25:42.041130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041083    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy"
Sep 15 12:25:42.041315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041182    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy"
Sep 15 12:25:42.041315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041191    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5777ecd8-a48b-4a45-9dc8-5c75aad99367" containerName="main"
Sep 15 12:25:42.041315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041199    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bb28ec-daad-4861-ba47-b1f002afb333" containerName="maas-api"
Sep 15 12:25:42.041315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.041206    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e452a9-1608-470d-9515-49169b37bf50" containerName="istio-proxy"
Sep 15 12:25:42.043523 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.043506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.046982 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.046961    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"auth-test-model-969130-kserve-self-signed-certs\""
Sep 15 12:25:42.047876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.047859    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"default-dockercfg-xf252\""
Sep 15 12:25:42.047970 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.047856    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"kube-root-ca.crt\""
Sep 15 12:25:42.047970 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.047902    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-969130\"/\"openshift-service-ca.crt\""
Sep 15 12:25:42.054363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.054339    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"]
Sep 15 12:25:42.081541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.081510    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:42.145655 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.145618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-model-cache\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.145807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.145688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c742e5c-9c17-4660-9688-f53b3690566d-tls-certs\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.145807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.145715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqnh\" (UniqueName: \"kubernetes.io/projected/4c742e5c-9c17-4660-9688-f53b3690566d-kube-api-access-2bqnh\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.145807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.145756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-tmp-dir\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.145807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.145778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-dshm\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.145807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.145803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-home\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-tmp-dir\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247312 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-dshm\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-home\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-model-cache\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c742e5c-9c17-4660-9688-f53b3690566d-tls-certs\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247549 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqnh\" (UniqueName: \"kubernetes.io/projected/4c742e5c-9c17-4660-9688-f53b3690566d-kube-api-access-2bqnh\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247823 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-tmp-dir\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-home\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.247888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.247842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-model-cache\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.249639 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.249616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-dshm\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.249957 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.249923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c742e5c-9c17-4660-9688-f53b3690566d-tls-certs\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.256099 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.256076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqnh\" (UniqueName: \"kubernetes.io/projected/4c742e5c-9c17-4660-9688-f53b3690566d-kube-api-access-2bqnh\") pod \"auth-test-model-969130-kserve-f56db8cbc-lzc5j\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") " pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.353268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.353228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:42.630950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:42.630914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"]
Sep 15 12:25:42.631249 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:25:42.631224    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c742e5c_9c17_4660_9688_f53b3690566d.slice/crio-3efa379d99802cca37b7ece98cace15fc4e8f6302696f94ee457c2a224da63d7 WatchSource:0}: Error finding container 3efa379d99802cca37b7ece98cace15fc4e8f6302696f94ee457c2a224da63d7: Status 404 returned error can't find the container with id 3efa379d99802cca37b7ece98cace15fc4e8f6302696f94ee457c2a224da63d7
Sep 15 12:25:43.519227 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:43.519189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j" event={"ID":"4c742e5c-9c17-4660-9688-f53b3690566d","Type":"ContainerStarted","Data":"9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180"}
Sep 15 12:25:43.519629 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:43.519240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j" event={"ID":"4c742e5c-9c17-4660-9688-f53b3690566d","Type":"ContainerStarted","Data":"3efa379d99802cca37b7ece98cace15fc4e8f6302696f94ee457c2a224da63d7"}
Sep 15 12:25:43.544161 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:43.544114    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j" podStartSLOduration=1.544099085 podStartE2EDuration="1.544099085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:43.5418797 +0000 UTC m=+1424.238439289" watchObservedRunningTime="2026-09-15 12:25:43.544099085 +0000 UTC m=+1424.240658960"
Sep 15 12:25:48.535012 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:48.534961    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:48.774968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:48.774924    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:48.871315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:48.871283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:52.354116 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:52.354001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:52.354116 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:52.354068    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:52.366772 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:52.366745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:52.582128 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:52.582095    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:25:53.537147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:25:53.537107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:09.687172 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:09.687131    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:13.172050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:13.171990    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.872337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:15.872293    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.964405 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:15.964375    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.217416 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.217322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.515355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.515265    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"]
Sep 15 12:26:16.519376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.519352    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.522325 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.522303    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f1e118-openshift-default-dockercfg-rxljg\""
Sep 15 12:26:16.538261 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.538236    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"]
Sep 15 12:26:16.553734 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.553707    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.595257 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-data\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.595459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-certs\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.595459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-envoy\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.595589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-podinfo\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.595589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfkq\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-kube-api-access-zsfkq\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.595589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-credential-socket\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.595589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-socket\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.596111 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-token\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.596111 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.595683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da032b7b-cb6e-4c88-862e-9e7fde782f65-istiod-ca-cert\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697102 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-podinfo\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfkq\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-kube-api-access-zsfkq\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-credential-socket\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-socket\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-token\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697526 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da032b7b-cb6e-4c88-862e-9e7fde782f65-istiod-ca-cert\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697526 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-data\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-certs\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-envoy\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-socket\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697804 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-credential-socket\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697804 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-data\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.697874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.697851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-certs\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.698085 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.698066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da032b7b-cb6e-4c88-862e-9e7fde782f65-istiod-ca-cert\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.699670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.699651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-envoy\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.699670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.699663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-podinfo\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.712101 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.712079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfkq\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-kube-api-access-zsfkq\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.716627 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.716605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-token\") pod \"e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") " pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:16.817276 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.817240    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.829786 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:16.829760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:17.107425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.107356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"]
Sep 15 12:26:17.110536 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:26:17.110501    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda032b7b_cb6e_4c88_862e_9e7fde782f65.slice/crio-4a412c81222c9be59aba7a49deffad853da7e567645dddd11d8abd12b58f621a WatchSource:0}: Error finding container 4a412c81222c9be59aba7a49deffad853da7e567645dddd11d8abd12b58f621a: Status 404 returned error can't find the container with id 4a412c81222c9be59aba7a49deffad853da7e567645dddd11d8abd12b58f621a
Sep 15 12:26:17.113138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.113097    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:26:17.113234 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.113185    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:26:17.113234 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.113226    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:26:17.713522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.713481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" event={"ID":"da032b7b-cb6e-4c88-862e-9e7fde782f65","Type":"ContainerStarted","Data":"9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"}
Sep 15 12:26:17.713693 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.713564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" event={"ID":"da032b7b-cb6e-4c88-862e-9e7fde782f65","Type":"ContainerStarted","Data":"4a412c81222c9be59aba7a49deffad853da7e567645dddd11d8abd12b58f621a"}
Sep 15 12:26:17.746571 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.746521    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podStartSLOduration=1.746504921 podStartE2EDuration="1.746504921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:17.743850739 +0000 UTC m=+1458.440410336" watchObservedRunningTime="2026-09-15 12:26:17.746504921 +0000 UTC m=+1458.443064499"
Sep 15 12:26:17.830497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:17.830467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:18.837442 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:18.837410    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:19.182296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:19.182206    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:19.726931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:19.726902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:19.727810 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:19.727788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:26:19.757377 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:19.757347    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:19.865697 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:19.865657    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:20.276676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:20.276644    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:21.230530 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:21.230484    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:22.228217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:22.228184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:23.149468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:23.149433    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:24.141361 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:24.141321    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:24.979473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:24.979442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:25.910607 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:25.910575    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"]
Sep 15 12:26:25.915078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:25.914581    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:25.919111 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:25.919081    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f1e118-serving-cert\""
Sep 15 12:26:25.938454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:25.938415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"]
Sep 15 12:26:25.956313 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:25.956177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:26.058996 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.058961    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:26.097661 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.097618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khhl\" (UniqueName: \"kubernetes.io/projected/57b72742-04c7-41c4-8496-d05861a7e838-kube-api-access-4khhl\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.097837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.097699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b72742-04c7-41c4-8496-d05861a7e838-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.097837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.097720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-metrics-certs\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.097837 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.097767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.199065 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.198941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khhl\" (UniqueName: \"kubernetes.io/projected/57b72742-04c7-41c4-8496-d05861a7e838-kube-api-access-4khhl\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.199264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.199111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b72742-04c7-41c4-8496-d05861a7e838-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.199264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.199146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-metrics-certs\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.199264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.199215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.199459 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:26:26.199404    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-serving-cert: secret "maas-api-e2e-api-a-f1e118-serving-cert" not found
Sep 15 12:26:26.199527 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:26:26.199512    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls podName:57b72742-04c7-41c4-8496-d05861a7e838 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:26.69948802 +0000 UTC m=+1467.396047575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls") pod "maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" (UID: "57b72742-04c7-41c4-8496-d05861a7e838") : secret "maas-api-e2e-api-a-f1e118-serving-cert" not found
Sep 15 12:26:26.199984 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.199958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b72742-04c7-41c4-8496-d05861a7e838-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.202467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.202422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-metrics-certs\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.212736 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.212703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khhl\" (UniqueName: \"kubernetes.io/projected/57b72742-04c7-41c4-8496-d05861a7e838-kube-api-access-4khhl\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.555686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.555588    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:26.705225 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.704995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.707717 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.707687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls\") pod \"maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:26.829231 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:26.829195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:27.112745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:27.112717    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"]
Sep 15 12:26:27.114872 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:26:27.114847    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b72742_04c7_41c4_8496_d05861a7e838.slice/crio-76ffad64a87013d3fd2ab1a9a3ad74a2f04d54fe4ab249745777fa913bbbe948 WatchSource:0}: Error finding container 76ffad64a87013d3fd2ab1a9a3ad74a2f04d54fe4ab249745777fa913bbbe948: Status 404 returned error can't find the container with id 76ffad64a87013d3fd2ab1a9a3ad74a2f04d54fe4ab249745777fa913bbbe948
Sep 15 12:26:27.116320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:27.116301    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:26:27.774534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:27.774447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" event={"ID":"57b72742-04c7-41c4-8496-d05861a7e838","Type":"ContainerStarted","Data":"ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff"}
Sep 15 12:26:27.774534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:27.774506    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:27.774534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:27.774523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" event={"ID":"57b72742-04c7-41c4-8496-d05861a7e838","Type":"ContainerStarted","Data":"76ffad64a87013d3fd2ab1a9a3ad74a2f04d54fe4ab249745777fa913bbbe948"}
Sep 15 12:26:27.795470 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:27.795417    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" podStartSLOduration=2.414270896 podStartE2EDuration="2.795399313s" totalImagesPullingTime="381.128417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:27.795305409 +0000 UTC m=+1468.491865001" watchObservedRunningTime="2026-09-15 12:26:27.795399313 +0000 UTC m=+1468.491958892"
Sep 15 12:26:29.646581 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:29.646540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:31.313180 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:31.313131    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.785770 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:33.785740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:26:49.574294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:49.574254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:50.326482 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:50.326434    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:51.440473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:51.440433    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:52.205887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:52.205852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:53.195944 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:53.195901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:54.242287 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:54.242256    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:54.849518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:54.849476    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:55.304602 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.304514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:55.410676 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.410637    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:55.673613 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.673580    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"]
Sep 15 12:26:55.677401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.677384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.680957 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.680936    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-eb9bbe-openshift-default-dockercfg-8rctm\""
Sep 15 12:26:55.696868 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.696845    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"]
Sep 15 12:26:55.745715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.745679    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:55.839652 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.839610    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-envoy\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.839917 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.839894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-token\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840095 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scvl\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-kube-api-access-5scvl\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-credential-socket\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840327 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-certs\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-data\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840530 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istiod-ca-cert\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840648 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-podinfo\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-socket\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.840943 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.840779    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:55.942100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.941998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-podinfo\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-socket\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-envoy\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-token\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scvl\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-kube-api-access-5scvl\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-credential-socket\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-certs\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-data\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.942338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istiod-ca-cert\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.943783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.942848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-socket\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.943783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.943092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istiod-ca-cert\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.943783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.943260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-data\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.943783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.943313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-certs\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.943783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.943569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-credential-socket\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.945288 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.945263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-podinfo\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.945754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.945728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-envoy\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.951922 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.951894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-token\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.955880 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.955851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scvl\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-kube-api-access-5scvl\") pod \"e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") " pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:55.988093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:55.988009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:56.299278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.299241    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"]
Sep 15 12:26:56.300757 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:26:56.300730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2417dd_b3f7_4e42_9012_7c99cc34ab71.slice/crio-1ae35378c53b5d7f2def747f9afdfddf8fc290e1790c3f46d8b53d527ad33674 WatchSource:0}: Error finding container 1ae35378c53b5d7f2def747f9afdfddf8fc290e1790c3f46d8b53d527ad33674: Status 404 returned error can't find the container with id 1ae35378c53b5d7f2def747f9afdfddf8fc290e1790c3f46d8b53d527ad33674
Sep 15 12:26:56.303294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.303261    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:26:56.303397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.303340    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:26:56.303397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.303384    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:26:56.438060 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.438000    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:56.954980 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.954944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" event={"ID":"3d2417dd-b3f7-4e42-9012-7c99cc34ab71","Type":"ContainerStarted","Data":"3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"}
Sep 15 12:26:56.955168 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.955008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" event={"ID":"3d2417dd-b3f7-4e42-9012-7c99cc34ab71","Type":"ContainerStarted","Data":"1ae35378c53b5d7f2def747f9afdfddf8fc290e1790c3f46d8b53d527ad33674"}
Sep 15 12:26:56.989150 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.989116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:56.997476 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:56.997428    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podStartSLOduration=1.9974124359999998 podStartE2EDuration="1.997412436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:56.995640378 +0000 UTC m=+1497.692199974" watchObservedRunningTime="2026-09-15 12:26:56.997412436 +0000 UTC m=+1497.693972014"
Sep 15 12:26:57.432757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:57.432720    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:57.995605 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:57.995575    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:58.543617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:58.543579    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:58.764889 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:58.764855    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:58.970171 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:58.970139    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:58.971282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:58.971258    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:26:59.152165 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:59.152132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:59.269727 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:59.269641    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:59.837644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:59.837606    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:59.855464 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:59.855440    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:26:59.857742 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:26:59.857718    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:27:00.837122 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:00.837069    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:01.718315 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.718278    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:01.989611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.989522    2583 generic.go:356] "Generic (PLEG): container finished" podID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerID="9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a" exitCode=0
Sep 15 12:27:01.990041 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.989599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" event={"ID":"da032b7b-cb6e-4c88-862e-9e7fde782f65","Type":"ContainerDied","Data":"9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"}
Sep 15 12:27:01.990220 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.990203    2583 scope.go:117] "RemoveContainer" containerID="9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"
Sep 15 12:27:01.991556 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.991527    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:27:01.991658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.991595    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:27:01.991658 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:01.991628    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:27:02.844518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:02.844477    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:02.996968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:02.996932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" event={"ID":"da032b7b-cb6e-4c88-862e-9e7fde782f65","Type":"ContainerStarted","Data":"9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23"}
Sep 15 12:27:03.832273 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:03.832224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:27:03.832617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:03.832593    2583 patch_prober.go:28] interesting pod/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 15 12:27:03.832804 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:03.832779    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 15 12:27:04.048308 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:04.048272    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:04.157591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:04.157504    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:04.727501 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:04.727458    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:27:04.831088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:04.831015    2583 patch_prober.go:28] interesting pod/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 15 12:27:04.831307 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:04.831139    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 15 12:27:05.233084 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:05.233046    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:05.830597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:05.830550    2583 patch_prober.go:28] interesting pod/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 15 12:27:05.830813 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:05.830635    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 15 12:27:06.269854 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:06.269763    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:06.830464 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:06.830428    2583 patch_prober.go:28] interesting pod/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 15 12:27:06.830745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:06.830716    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 15 12:27:07.401665 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.401631    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"]
Sep 15 12:27:07.404589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.404568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.410319 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.410291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-eb9bbe-serving-cert\""
Sep 15 12:27:07.422688 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.422661    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"]
Sep 15 12:27:07.457714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.457652    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:07.462978 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.462941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.463151 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.463119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-metrics-certs\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.463219 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.463165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/32270fc2-1c0e-4f11-8741-bd4877bb0283-kube-api-access-xqfc4\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.463322 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.463290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32270fc2-1c0e-4f11-8741-bd4877bb0283-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.549649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.549601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:07.564971 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.564931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-metrics-certs\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.565148 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.565068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/32270fc2-1c0e-4f11-8741-bd4877bb0283-kube-api-access-xqfc4\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.565199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.565150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32270fc2-1c0e-4f11-8741-bd4877bb0283-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.565258 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.565242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.565409 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:27:07.565386    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-serving-cert: secret "maas-api-e2e-api-b-eb9bbe-serving-cert" not found
Sep 15 12:27:07.565475 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:27:07.565463    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls podName:32270fc2-1c0e-4f11-8741-bd4877bb0283 nodeName:}" failed. No retries permitted until 2026-09-15 12:27:08.065443669 +0000 UTC m=+1508.762003225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls") pod "maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" (UID: "32270fc2-1c0e-4f11-8741-bd4877bb0283") : secret "maas-api-e2e-api-b-eb9bbe-serving-cert" not found
Sep 15 12:27:07.565856 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.565833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32270fc2-1c0e-4f11-8741-bd4877bb0283-openshift-service-ca\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.567520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.567501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-metrics-certs\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.585888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.585860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/32270fc2-1c0e-4f11-8741-bd4877bb0283-kube-api-access-xqfc4\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:07.830851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.830801    2583 patch_prober.go:28] interesting pod/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Sep 15 12:27:07.831091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.830874    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Sep 15 12:27:07.910750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:07.910705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:08.071364 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.071327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:08.073757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.073717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:08.073913 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.073894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls\") pod \"maas-api-e2e-api-b-eb9bbe-d77474598-lrk75\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:08.166746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.166648    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:08.344853 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.344807    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:08.641855 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.641823    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"]
Sep 15 12:27:08.643126 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:27:08.643093    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32270fc2_1c0e_4f11_8741_bd4877bb0283.slice/crio-da1861daf834e9666fa8923ec31f892249991643b28528f5f1ae329aaeb69a27 WatchSource:0}: Error finding container da1861daf834e9666fa8923ec31f892249991643b28528f5f1ae329aaeb69a27: Status 404 returned error can't find the container with id da1861daf834e9666fa8923ec31f892249991643b28528f5f1ae329aaeb69a27
Sep 15 12:27:08.740527 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.740416    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:08.835264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.835230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:27:08.836237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:08.836219    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:27:09.050146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:09.050055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" event={"ID":"32270fc2-1c0e-4f11-8741-bd4877bb0283","Type":"ContainerStarted","Data":"da1861daf834e9666fa8923ec31f892249991643b28528f5f1ae329aaeb69a27"}
Sep 15 12:27:09.664555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:09.664514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:10.057396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:10.057302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" event={"ID":"32270fc2-1c0e-4f11-8741-bd4877bb0283","Type":"ContainerStarted","Data":"359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa"}
Sep 15 12:27:10.057552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:10.057407    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:10.083366 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:10.083306    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" podStartSLOduration=2.653582733 podStartE2EDuration="3.083289322s" totalImagesPullingTime="429.706589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:27:10.079930735 +0000 UTC m=+1510.776490314" watchObservedRunningTime="2026-09-15 12:27:10.083289322 +0000 UTC m=+1510.779848896"
Sep 15 12:27:10.523307 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:10.523272    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:11.517604 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:11.517565    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:16.068121 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:16.068086    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:27:17.235276 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.235243    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"]
Sep 15 12:27:17.235845 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.235454    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn" podUID="83a5da10-3ddd-4880-8d47-001c454ca14a" containerName="maas-api" containerID="cri-o://a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2" gracePeriod=30
Sep 15 12:27:17.321032 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.320983    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:17.440715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.440678    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:17.539714 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.539690    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:27:17.596826 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.596768    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:17.671016 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.670984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-maas-api-tls\") pod \"83a5da10-3ddd-4880-8d47-001c454ca14a\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") "
Sep 15 12:27:17.671016 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.671019    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-metrics-certs\") pod \"83a5da10-3ddd-4880-8d47-001c454ca14a\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") "
Sep 15 12:27:17.671320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.671121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxmln\" (UniqueName: \"kubernetes.io/projected/83a5da10-3ddd-4880-8d47-001c454ca14a-kube-api-access-pxmln\") pod \"83a5da10-3ddd-4880-8d47-001c454ca14a\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") "
Sep 15 12:27:17.671320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.671191    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a5da10-3ddd-4880-8d47-001c454ca14a-openshift-service-ca\") pod \"83a5da10-3ddd-4880-8d47-001c454ca14a\" (UID: \"83a5da10-3ddd-4880-8d47-001c454ca14a\") "
Sep 15 12:27:17.671611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.671583    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a5da10-3ddd-4880-8d47-001c454ca14a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83a5da10-3ddd-4880-8d47-001c454ca14a" (UID: "83a5da10-3ddd-4880-8d47-001c454ca14a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:27:17.673265 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.673232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83a5da10-3ddd-4880-8d47-001c454ca14a" (UID: "83a5da10-3ddd-4880-8d47-001c454ca14a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:17.673368 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.673279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83a5da10-3ddd-4880-8d47-001c454ca14a" (UID: "83a5da10-3ddd-4880-8d47-001c454ca14a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:17.673368 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.673311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a5da10-3ddd-4880-8d47-001c454ca14a-kube-api-access-pxmln" (OuterVolumeSpecName: "kube-api-access-pxmln") pod "83a5da10-3ddd-4880-8d47-001c454ca14a" (UID: "83a5da10-3ddd-4880-8d47-001c454ca14a"). InnerVolumeSpecName "kube-api-access-pxmln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:17.773061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.772964    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:17.773061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.773002    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a5da10-3ddd-4880-8d47-001c454ca14a-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:17.773061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.773017    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmln\" (UniqueName: \"kubernetes.io/projected/83a5da10-3ddd-4880-8d47-001c454ca14a-kube-api-access-pxmln\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:17.773061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.773048    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a5da10-3ddd-4880-8d47-001c454ca14a-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:17.835282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:17.835247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:18.062048 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.061944    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:18.104723 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.104691    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerID="70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616" exitCode=137
Sep 15 12:27:18.104885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.104759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" event={"ID":"d2341045-6356-4333-b9fb-e3a7adfdbc14","Type":"ContainerDied","Data":"70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"}
Sep 15 12:27:18.105197 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.105177    2583 scope.go:117] "RemoveContainer" containerID="70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"
Sep 15 12:27:18.106403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.106339    2583 generic.go:356] "Generic (PLEG): container finished" podID="83a5da10-3ddd-4880-8d47-001c454ca14a" containerID="a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2" exitCode=0
Sep 15 12:27:18.106403 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.106381    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"
Sep 15 12:27:18.106552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.106375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn" event={"ID":"83a5da10-3ddd-4880-8d47-001c454ca14a","Type":"ContainerDied","Data":"a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2"}
Sep 15 12:27:18.106552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.106506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn" event={"ID":"83a5da10-3ddd-4880-8d47-001c454ca14a","Type":"ContainerDied","Data":"ea8e799510ec9301c1d83371dc58ebac3c195e8c6e61879a93744fdb3ce8bc1f"}
Sep 15 12:27:18.106552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.106542    2583 scope.go:117] "RemoveContainer" containerID="a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2"
Sep 15 12:27:18.123267 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.123243    2583 scope.go:117] "RemoveContainer" containerID="a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2"
Sep 15 12:27:18.123641 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:27:18.123612    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2\": container with ID starting with a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2 not found: ID does not exist" containerID="a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2"
Sep 15 12:27:18.123711 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.123651    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2"} err="failed to get container status \"a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2\": rpc error: code = NotFound desc = could not find container \"a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2\": container with ID starting with a4d6171f6e9b47ec6fd29614a00bf851802d2d027549beaf4ffd7dcf7e5b8db2 not found: ID does not exist"
Sep 15 12:27:18.163121 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.163090    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"]
Sep 15 12:27:18.171452 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.171427    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-680229-6579bb9647-g7xjn"]
Sep 15 12:27:18.265452 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.265419    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:18.478606 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:18.478569    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:19.115264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:19.115229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" event={"ID":"d2341045-6356-4333-b9fb-e3a7adfdbc14","Type":"ContainerStarted","Data":"94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b"}
Sep 15 12:27:19.115479 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:19.115452    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:27:19.809154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:19.809120    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a5da10-3ddd-4880-8d47-001c454ca14a" path="/var/lib/kubelet/pods/83a5da10-3ddd-4880-8d47-001c454ca14a/volumes"
Sep 15 12:27:19.896741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:19.896692    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:21.328662 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:21.328625    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:22.829808 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:22.829719    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:23.853363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:23.853329    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:23.963751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:23.963715    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:24.821697 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:24.821657    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:25.818609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:25.818576    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:26.386765 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:26.386731    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:26.486888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:26.486853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:26.609905 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:26.609866    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:26.708874 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:26.708775    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:27.603406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:27.603368    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:28.384703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:28.384671    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:30.125223 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:30.125194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:27:31.868040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:31.867971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:32.007864 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:32.007820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:32.202038 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:32.201930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:33.212001 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:33.211963    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:34.076571 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:34.076531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:37.702613 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:37.702575    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:38.998672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:38.998628    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:41.252491 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:41.252455    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:41.352730 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:41.352698    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:41.873214 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:41.873184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:42.711928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:42.711892    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"]
Sep 15 12:27:42.712950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:42.712908    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" podUID="57b72742-04c7-41c4-8496-d05861a7e838" containerName="maas-api" containerID="cri-o://ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff" gracePeriod=30
Sep 15 12:27:42.749147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:42.749114    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:42.840577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:42.840527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:42.939566 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:42.939536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:42.980809 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:42.980785    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:27:43.008559 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.008527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:43.044433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.044403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b72742-04c7-41c4-8496-d05861a7e838-openshift-service-ca\") pod \"57b72742-04c7-41c4-8496-d05861a7e838\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") "
Sep 15 12:27:43.044642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.044537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4khhl\" (UniqueName: \"kubernetes.io/projected/57b72742-04c7-41c4-8496-d05861a7e838-kube-api-access-4khhl\") pod \"57b72742-04c7-41c4-8496-d05861a7e838\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") "
Sep 15 12:27:43.044642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.044602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-metrics-certs\") pod \"57b72742-04c7-41c4-8496-d05861a7e838\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") "
Sep 15 12:27:43.044642 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.044632    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls\") pod \"57b72742-04c7-41c4-8496-d05861a7e838\" (UID: \"57b72742-04c7-41c4-8496-d05861a7e838\") "
Sep 15 12:27:43.044824 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.044789    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b72742-04c7-41c4-8496-d05861a7e838-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57b72742-04c7-41c4-8496-d05861a7e838" (UID: "57b72742-04c7-41c4-8496-d05861a7e838"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:27:43.045066 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.045041    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b72742-04c7-41c4-8496-d05861a7e838-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:43.046833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.046807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57b72742-04c7-41c4-8496-d05861a7e838" (UID: "57b72742-04c7-41c4-8496-d05861a7e838"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:43.047056 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.046999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57b72742-04c7-41c4-8496-d05861a7e838" (UID: "57b72742-04c7-41c4-8496-d05861a7e838"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:27:43.047193 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.047073    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b72742-04c7-41c4-8496-d05861a7e838-kube-api-access-4khhl" (OuterVolumeSpecName: "kube-api-access-4khhl") pod "57b72742-04c7-41c4-8496-d05861a7e838" (UID: "57b72742-04c7-41c4-8496-d05861a7e838"). InnerVolumeSpecName "kube-api-access-4khhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:43.146271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.146226    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khhl\" (UniqueName: \"kubernetes.io/projected/57b72742-04c7-41c4-8496-d05861a7e838-kube-api-access-4khhl\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:43.146271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.146269    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:43.146550 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.146285    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b72742-04c7-41c4-8496-d05861a7e838-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:43.277286 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.277191    2583 generic.go:356] "Generic (PLEG): container finished" podID="57b72742-04c7-41c4-8496-d05861a7e838" containerID="ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff" exitCode=0
Sep 15 12:27:43.277286 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.277244    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"
Sep 15 12:27:43.277286 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.277261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" event={"ID":"57b72742-04c7-41c4-8496-d05861a7e838","Type":"ContainerDied","Data":"ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff"}
Sep 15 12:27:43.277522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.277342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth" event={"ID":"57b72742-04c7-41c4-8496-d05861a7e838","Type":"ContainerDied","Data":"76ffad64a87013d3fd2ab1a9a3ad74a2f04d54fe4ab249745777fa913bbbe948"}
Sep 15 12:27:43.277522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.277378    2583 scope.go:117] "RemoveContainer" containerID="ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff"
Sep 15 12:27:43.288932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.288909    2583 scope.go:117] "RemoveContainer" containerID="ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff"
Sep 15 12:27:43.289310 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:27:43.289289    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff\": container with ID starting with ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff not found: ID does not exist" containerID="ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff"
Sep 15 12:27:43.289376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.289318    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff"} err="failed to get container status \"ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff\": rpc error: code = NotFound desc = could not find container \"ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff\": container with ID starting with ac94be852141aadb2bf91dad4caeec61187061dd66ce22fc6d2948e18dfb9eff not found: ID does not exist"
Sep 15 12:27:43.302537 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.302512    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"]
Sep 15 12:27:43.306825 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.306801    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1e118-5d4f48fbf4-k9zth"]
Sep 15 12:27:43.326667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.326635    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:43.496869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.496819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:43.815380 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:43.815345    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b72742-04c7-41c4-8496-d05861a7e838" path="/var/lib/kubelet/pods/57b72742-04c7-41c4-8496-d05861a7e838/volumes"
Sep 15 12:27:44.411008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:44.410965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:45.220081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:45.220042    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:46.128694 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:46.128651    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:46.944084 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:46.944045    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:50.389768 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:50.389733    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"]
Sep 15 12:27:50.390282 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:50.389970    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" containerID="cri-o://9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23" gracePeriod=30
Sep 15 12:27:50.566728 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:50.566691    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:50.677849 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:50.677752    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:51.108787 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:51.108751    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:52.018775 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:52.018735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:52.894453 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:52.894412    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:53.838230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:53.838191    2583 patch_prober.go:28] interesting pod/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:27:53.838695 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:53.838282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:27:53.925931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:53.925892    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:55.730210 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.729841    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:27:55.778218 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778189    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-envoy\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da032b7b-cb6e-4c88-862e-9e7fde782f65-istiod-ca-cert\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778287    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-certs\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778322    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-podinfo\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-token\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778425 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778408    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-data\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778460    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-socket\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-credential-socket\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778562    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsfkq\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-kube-api-access-zsfkq\") pod \"da032b7b-cb6e-4c88-862e-9e7fde782f65\" (UID: \"da032b7b-cb6e-4c88-862e-9e7fde782f65\") "
Sep 15 12:27:55.778687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:55.778687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778654    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da032b7b-cb6e-4c88-862e-9e7fde782f65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:27:55.778934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778798    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-data" (OuterVolumeSpecName: "istio-data") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:55.778934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:55.779093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.778965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:55.779093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.779017    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.779093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.779066    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.779093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.779082    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da032b7b-cb6e-4c88-862e-9e7fde782f65-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.779313 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.779096    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.780954 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.780924    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:27:55.781231 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.781129    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-token" (OuterVolumeSpecName: "istio-token") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:55.781231 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.781153    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:27:55.781231 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.781198    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-kube-api-access-zsfkq" (OuterVolumeSpecName: "kube-api-access-zsfkq") pod "da032b7b-cb6e-4c88-862e-9e7fde782f65" (UID: "da032b7b-cb6e-4c88-862e-9e7fde782f65"). InnerVolumeSpecName "kube-api-access-zsfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:27:55.879846 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.879816    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.879846 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.879843    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.879846 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.879852    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.880138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.879862    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da032b7b-cb6e-4c88-862e-9e7fde782f65-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:55.880138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:55.879872    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsfkq\" (UniqueName: \"kubernetes.io/projected/da032b7b-cb6e-4c88-862e-9e7fde782f65-kube-api-access-zsfkq\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:27:56.366718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.366685    2583 generic.go:356] "Generic (PLEG): container finished" podID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerID="9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23" exitCode=0
Sep 15 12:27:56.366888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.366729    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"
Sep 15 12:27:56.366888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.366766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" event={"ID":"da032b7b-cb6e-4c88-862e-9e7fde782f65","Type":"ContainerDied","Data":"9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23"}
Sep 15 12:27:56.366888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.366819    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92" event={"ID":"da032b7b-cb6e-4c88-862e-9e7fde782f65","Type":"ContainerDied","Data":"4a412c81222c9be59aba7a49deffad853da7e567645dddd11d8abd12b58f621a"}
Sep 15 12:27:56.366888 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.366842    2583 scope.go:117] "RemoveContainer" containerID="9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23"
Sep 15 12:27:56.376615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.376598    2583 scope.go:117] "RemoveContainer" containerID="9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"
Sep 15 12:27:56.385085 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.385066    2583 scope.go:117] "RemoveContainer" containerID="9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23"
Sep 15 12:27:56.385790 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:27:56.385749    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23\": container with ID starting with 9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23 not found: ID does not exist" containerID="9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23"
Sep 15 12:27:56.385876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.385789    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23"} err="failed to get container status \"9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23\": rpc error: code = NotFound desc = could not find container \"9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23\": container with ID starting with 9bf30c168b45b93680241e79efd6875004a1c6005bcb352dabcaa532e0651a23 not found: ID does not exist"
Sep 15 12:27:56.385876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.385808    2583 scope.go:117] "RemoveContainer" containerID="9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"
Sep 15 12:27:56.386133 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:27:56.386108    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a\": container with ID starting with 9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a not found: ID does not exist" containerID="9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"
Sep 15 12:27:56.386230 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.386136    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a"} err="failed to get container status \"9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a\": rpc error: code = NotFound desc = could not find container \"9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a\": container with ID starting with 9b205c3b58504fe7ba62cb904d21863bf61ac741bcc8e78137ad1bc6d3a2ba0a not found: ID does not exist"
Sep 15 12:27:56.388139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.388116    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"]
Sep 15 12:27:56.391541 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:56.391523    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f1e118-openshift-default-6cb8bc7484-gpv92"]
Sep 15 12:27:57.779542 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:57.779511    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:57.808232 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:57.808190    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" path="/var/lib/kubelet/pods/da032b7b-cb6e-4c88-862e-9e7fde782f65/volumes"
Sep 15 12:27:57.881667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:57.881633    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:58.433477 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:27:58.433444    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:00.012241 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:00.012199    2583 patch_prober.go:28] interesting pod/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 12:28:00.012640 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:00.012270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 12:28:00.910059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:00.910002    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:02.404892 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:02.404861    2583 generic.go:356] "Generic (PLEG): container finished" podID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerID="3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348" exitCode=0
Sep 15 12:28:02.405379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:02.404933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" event={"ID":"3d2417dd-b3f7-4e42-9012-7c99cc34ab71","Type":"ContainerDied","Data":"3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"}
Sep 15 12:28:02.405433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:02.405400    2583 scope.go:117] "RemoveContainer" containerID="3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"
Sep 15 12:28:02.406597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:02.406569    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:28:02.406663 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:02.406651    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:28:02.406706 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:02.406684    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:28:03.431551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:03.431494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" event={"ID":"3d2417dd-b3f7-4e42-9012-7c99cc34ab71","Type":"ContainerStarted","Data":"431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7"}
Sep 15 12:28:03.988807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:03.988774    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:28:03.988985 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:03.988959    2583 patch_prober.go:28] interesting pod/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 15 12:28:03.989072 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:03.989035    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 15 12:28:04.988520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:04.988482    2583 patch_prober.go:28] interesting pod/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 15 12:28:04.988926 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:04.988543    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 15 12:28:05.989289 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:05.989255    2583 patch_prober.go:28] interesting pod/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 15 12:28:05.989757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:05.989315    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 15 12:28:06.551089 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.551055    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"]
Sep 15 12:28:06.551554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.551520    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" podUID="32270fc2-1c0e-4f11-8741-bd4877bb0283" containerName="maas-api" containerID="cri-o://359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa" gracePeriod=30
Sep 15 12:28:06.584538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.584503    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:06.684981 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.684903    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:06.851657 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.851619    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:06.860891 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.860858    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:28:06.946486 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.946441    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:06.989370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.989334    2583 patch_prober.go:28] interesting pod/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 15 12:28:06.989770 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:06.989390    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 15 12:28:07.002089 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.002057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls\") pod \"32270fc2-1c0e-4f11-8741-bd4877bb0283\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") "
Sep 15 12:28:07.002250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.002114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-metrics-certs\") pod \"32270fc2-1c0e-4f11-8741-bd4877bb0283\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") "
Sep 15 12:28:07.002250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.002162    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32270fc2-1c0e-4f11-8741-bd4877bb0283-openshift-service-ca\") pod \"32270fc2-1c0e-4f11-8741-bd4877bb0283\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") "
Sep 15 12:28:07.002250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.002206    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/32270fc2-1c0e-4f11-8741-bd4877bb0283-kube-api-access-xqfc4\") pod \"32270fc2-1c0e-4f11-8741-bd4877bb0283\" (UID: \"32270fc2-1c0e-4f11-8741-bd4877bb0283\") "
Sep 15 12:28:07.002595 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.002553    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32270fc2-1c0e-4f11-8741-bd4877bb0283-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32270fc2-1c0e-4f11-8741-bd4877bb0283" (UID: "32270fc2-1c0e-4f11-8741-bd4877bb0283"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:07.004496 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.004464    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32270fc2-1c0e-4f11-8741-bd4877bb0283-kube-api-access-xqfc4" (OuterVolumeSpecName: "kube-api-access-xqfc4") pod "32270fc2-1c0e-4f11-8741-bd4877bb0283" (UID: "32270fc2-1c0e-4f11-8741-bd4877bb0283"). InnerVolumeSpecName "kube-api-access-xqfc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:07.004698 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.004681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32270fc2-1c0e-4f11-8741-bd4877bb0283" (UID: "32270fc2-1c0e-4f11-8741-bd4877bb0283"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:07.004749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.004708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32270fc2-1c0e-4f11-8741-bd4877bb0283" (UID: "32270fc2-1c0e-4f11-8741-bd4877bb0283"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:07.103597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.103494    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:07.103597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.103528    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32270fc2-1c0e-4f11-8741-bd4877bb0283-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:07.103597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.103545    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32270fc2-1c0e-4f11-8741-bd4877bb0283-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:07.103597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.103559    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfc4\" (UniqueName: \"kubernetes.io/projected/32270fc2-1c0e-4f11-8741-bd4877bb0283-kube-api-access-xqfc4\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:07.244968 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.244926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:07.416927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.416795    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:07.458142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.458110    2583 generic.go:356] "Generic (PLEG): container finished" podID="32270fc2-1c0e-4f11-8741-bd4877bb0283" containerID="359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa" exitCode=0
Sep 15 12:28:07.458314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.458156    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"
Sep 15 12:28:07.458314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.458160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" event={"ID":"32270fc2-1c0e-4f11-8741-bd4877bb0283","Type":"ContainerDied","Data":"359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa"}
Sep 15 12:28:07.458314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.458294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75" event={"ID":"32270fc2-1c0e-4f11-8741-bd4877bb0283","Type":"ContainerDied","Data":"da1861daf834e9666fa8923ec31f892249991643b28528f5f1ae329aaeb69a27"}
Sep 15 12:28:07.458426 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.458325    2583 scope.go:117] "RemoveContainer" containerID="359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa"
Sep 15 12:28:07.471621 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.471599    2583 scope.go:117] "RemoveContainer" containerID="359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa"
Sep 15 12:28:07.471935 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:07.471911    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa\": container with ID starting with 359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa not found: ID does not exist" containerID="359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa"
Sep 15 12:28:07.472008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.471950    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa"} err="failed to get container status \"359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa\": rpc error: code = NotFound desc = could not find container \"359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa\": container with ID starting with 359b08de759dc7c324c5ad0fda55ec98c46f1bf2c5b7158a3afd172383bcfffa not found: ID does not exist"
Sep 15 12:28:07.484751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.484724    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"]
Sep 15 12:28:07.492331 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.492308    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eb9bbe-d77474598-lrk75"]
Sep 15 12:28:07.517764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.517686    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:07.809782 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.809686    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32270fc2-1c0e-4f11-8741-bd4877bb0283" path="/var/lib/kubelet/pods/32270fc2-1c0e-4f11-8741-bd4877bb0283/volumes"
Sep 15 12:28:07.989159 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.989125    2583 patch_prober.go:28] interesting pod/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 15 12:28:07.989304 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:07.989179    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 15 12:28:08.256932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:08.256894    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:08.992332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:08.992303    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:28:08.992596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:08.992576    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:28:08.993385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:08.993366    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:28:09.462817 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:09.462775    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:10.407481 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:10.407428    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:11.148470 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:11.148433    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:12.088150 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:12.088120    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:13.932884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:13.932849    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:14.042787 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:14.042751    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:14.201099 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:14.200979    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:14.201932 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:14.201907    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"]
Sep 15 12:28:14.202146 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:14.202123    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy" containerID="cri-o://431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7" gracePeriod=30
Sep 15 12:28:14.874114 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:14.874075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:15.785710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:15.785671    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:16.730864 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:16.730831    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:17.816508 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:17.816474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:18.567141 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:18.567105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:19.310045 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.309993    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:19.466073 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.466002    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:28:19.537143 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537116    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-certs\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537310 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-socket\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537310 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-credential-socket\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537310 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537249    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-podinfo\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537310 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537289    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-token\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istiod-ca-cert\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5scvl\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-kube-api-access-5scvl\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-envoy\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537442    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:19.537555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-data\") pod \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\" (UID: \"3d2417dd-b3f7-4e42-9012-7c99cc34ab71\") "
Sep 15 12:28:19.537555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537550    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:19.537865 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537564    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:19.537931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537890    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.537931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537912    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.537931 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.537925    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.538149 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.538127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:19.538303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.538200    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:19.540097 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.540045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-kube-api-access-5scvl" (OuterVolumeSpecName: "kube-api-access-5scvl") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "kube-api-access-5scvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:19.540233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.540193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:19.540348 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.540291    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:19.540563 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.540533    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d2417dd-b3f7-4e42-9012-7c99cc34ab71" (UID: "3d2417dd-b3f7-4e42-9012-7c99cc34ab71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:28:19.547087 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.547010    2583 generic.go:356] "Generic (PLEG): container finished" podID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerID="431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7" exitCode=0
Sep 15 12:28:19.547181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.547116    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"
Sep 15 12:28:19.547181 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.547142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" event={"ID":"3d2417dd-b3f7-4e42-9012-7c99cc34ab71","Type":"ContainerDied","Data":"431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7"}
Sep 15 12:28:19.547299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.547205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9" event={"ID":"3d2417dd-b3f7-4e42-9012-7c99cc34ab71","Type":"ContainerDied","Data":"1ae35378c53b5d7f2def747f9afdfddf8fc290e1790c3f46d8b53d527ad33674"}
Sep 15 12:28:19.547299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.547238    2583 scope.go:117] "RemoveContainer" containerID="431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7"
Sep 15 12:28:19.566133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.566100    2583 scope.go:117] "RemoveContainer" containerID="3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"
Sep 15 12:28:19.574782 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.574762    2583 scope.go:117] "RemoveContainer" containerID="431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7"
Sep 15 12:28:19.575100 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:19.575078    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7\": container with ID starting with 431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7 not found: ID does not exist" containerID="431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7"
Sep 15 12:28:19.575158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.575107    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7"} err="failed to get container status \"431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7\": rpc error: code = NotFound desc = could not find container \"431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7\": container with ID starting with 431d6f2a4cd930c8f68ca60e8d5d7228ce29ea204fe86940a35ec41a474f51f7 not found: ID does not exist"
Sep 15 12:28:19.575158 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.575125    2583 scope.go:117] "RemoveContainer" containerID="3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"
Sep 15 12:28:19.575390 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:19.575372    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348\": container with ID starting with 3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348 not found: ID does not exist" containerID="3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"
Sep 15 12:28:19.575436 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.575395    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348"} err="failed to get container status \"3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348\": rpc error: code = NotFound desc = could not find container \"3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348\": container with ID starting with 3fabb546fcfd8f1cd88a56c3aa2fe49550c32233c223b58a54336db784874348 not found: ID does not exist"
Sep 15 12:28:19.601783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.601756    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"]
Sep 15 12:28:19.607332 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.607306    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-eb9bbe-openshift-default-69d7d9b964-xztv9"]
Sep 15 12:28:19.639235 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.639213    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.639235 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.639234    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.639396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.639244    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.639396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.639253    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scvl\" (UniqueName: \"kubernetes.io/projected/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-kube-api-access-5scvl\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.639396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.639261    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.639396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.639270    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2417dd-b3f7-4e42-9012-7c99cc34ab71-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:19.807921 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:19.807829    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" path="/var/lib/kubelet/pods/3d2417dd-b3f7-4e42-9012-7c99cc34ab71/volumes"
Sep 15 12:28:20.216338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:20.216299    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:25.175011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.174979    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"]
Sep 15 12:28:25.175554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.175257    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt" podUID="deb3fea2-df4e-472f-9f34-11ec2a62bbfa" containerName="maas-api" containerID="cri-o://a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6" gracePeriod=30
Sep 15 12:28:25.216593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.216557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:25.329949 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.329910    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:25.429451 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.429379    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:25.430798 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.430780    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:28:25.498499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.497361    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69hng\" (UniqueName: \"kubernetes.io/projected/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-kube-api-access-69hng\") pod \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") "
Sep 15 12:28:25.498499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.497471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-metrics-certs\") pod \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") "
Sep 15 12:28:25.498499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.497531    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-maas-api-tls\") pod \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") "
Sep 15 12:28:25.498499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.497580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-openshift-service-ca\") pod \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\" (UID: \"deb3fea2-df4e-472f-9f34-11ec2a62bbfa\") "
Sep 15 12:28:25.499081 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.499014    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "deb3fea2-df4e-472f-9f34-11ec2a62bbfa" (UID: "deb3fea2-df4e-472f-9f34-11ec2a62bbfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:25.500783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.500752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "deb3fea2-df4e-472f-9f34-11ec2a62bbfa" (UID: "deb3fea2-df4e-472f-9f34-11ec2a62bbfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:25.500893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.500874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-kube-api-access-69hng" (OuterVolumeSpecName: "kube-api-access-69hng") pod "deb3fea2-df4e-472f-9f34-11ec2a62bbfa" (UID: "deb3fea2-df4e-472f-9f34-11ec2a62bbfa"). InnerVolumeSpecName "kube-api-access-69hng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:25.501680 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.501644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "deb3fea2-df4e-472f-9f34-11ec2a62bbfa" (UID: "deb3fea2-df4e-472f-9f34-11ec2a62bbfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:25.527977 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.527942    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:25.584974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.584935    2583 generic.go:356] "Generic (PLEG): container finished" podID="deb3fea2-df4e-472f-9f34-11ec2a62bbfa" containerID="a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6" exitCode=0
Sep 15 12:28:25.585199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.584984    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"
Sep 15 12:28:25.585199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.585017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt" event={"ID":"deb3fea2-df4e-472f-9f34-11ec2a62bbfa","Type":"ContainerDied","Data":"a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6"}
Sep 15 12:28:25.585199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.585121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt" event={"ID":"deb3fea2-df4e-472f-9f34-11ec2a62bbfa","Type":"ContainerDied","Data":"051f53513d3e56ee96c8ec10ec0cc8785afe6f83c63fffa96a0cb20e67037a1c"}
Sep 15 12:28:25.585199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.585153    2583 scope.go:117] "RemoveContainer" containerID="a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6"
Sep 15 12:28:25.596743 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.596718    2583 scope.go:117] "RemoveContainer" containerID="a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6"
Sep 15 12:28:25.597137 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:25.597113    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6\": container with ID starting with a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6 not found: ID does not exist" containerID="a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6"
Sep 15 12:28:25.597224 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.597146    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6"} err="failed to get container status \"a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6\": rpc error: code = NotFound desc = could not find container \"a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6\": container with ID starting with a35c073a93214840f09d1b7f795419f3b7812e6dde5627d227d50662d22c02c6 not found: ID does not exist"
Sep 15 12:28:25.598616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.598586    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:25.598616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.598609    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:25.598749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.598623    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:25.598749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.598637    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hng\" (UniqueName: \"kubernetes.io/projected/deb3fea2-df4e-472f-9f34-11ec2a62bbfa-kube-api-access-69hng\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:25.618360 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.618318    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"]
Sep 15 12:28:25.631384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.631301    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-58b0fd-5bc9dfc574-f5mwt"]
Sep 15 12:28:25.807344 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:25.807255    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb3fea2-df4e-472f-9f34-11ec2a62bbfa" path="/var/lib/kubelet/pods/deb3fea2-df4e-472f-9f34-11ec2a62bbfa/volumes"
Sep 15 12:28:28.977512 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:28.977468    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:29.078307 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:29.078268    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:30.020271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:30.020227    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:33.306915 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:33.306869    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"]
Sep 15 12:28:33.307401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:33.307191    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy" containerID="cri-o://35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53" gracePeriod=30
Sep 15 12:28:34.722133 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:34.722100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:34.824916 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:34.824880    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:34.925168 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:34.925130    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:35.811611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:35.811575    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:36.724424 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:36.724391    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:36.840401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:36.840367    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:37.859438 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:37.859401    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:38.568763 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.568737    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:28:38.633112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633002    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-socket\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-data\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633151    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-envoy\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-podinfo\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633209    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istiod-ca-cert\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633241    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr9l2\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-kube-api-access-xr9l2\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:38.633284 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633275    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-credential-socket\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633324    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-token\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633381    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-certs\") pod \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\" (UID: \"3b54dcf7-1f29-49bd-bfd5-2650deeccda7\") "
Sep 15 12:28:38.633683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633517    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:38.633843 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633774    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:38.633843 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633803    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:38.633985 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.633964    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.634195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.634178    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.634328 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.634313    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.634434 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.634423    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.634510 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.634373    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:38.636195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.636163    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:38.636309 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.636192    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-kube-api-access-xr9l2" (OuterVolumeSpecName: "kube-api-access-xr9l2") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "kube-api-access-xr9l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:38.636309 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.636256    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:38.636438 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.636357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b54dcf7-1f29-49bd-bfd5-2650deeccda7" (UID: "3b54dcf7-1f29-49bd-bfd5-2650deeccda7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:28:38.662649 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.662613    2583 generic.go:356] "Generic (PLEG): container finished" podID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerID="35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53" exitCode=0
Sep 15 12:28:38.662793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.662658    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"
Sep 15 12:28:38.662793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.662657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" event={"ID":"3b54dcf7-1f29-49bd-bfd5-2650deeccda7","Type":"ContainerDied","Data":"35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53"}
Sep 15 12:28:38.662793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.662746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s" event={"ID":"3b54dcf7-1f29-49bd-bfd5-2650deeccda7","Type":"ContainerDied","Data":"aab11b9f57b86fc0aa571af7a7df20adba8af307464e1690420779aa4835868b"}
Sep 15 12:28:38.662793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.662775    2583 scope.go:117] "RemoveContainer" containerID="35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53"
Sep 15 12:28:38.680244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.680222    2583 scope.go:117] "RemoveContainer" containerID="e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"
Sep 15 12:28:38.688982 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.688962    2583 scope.go:117] "RemoveContainer" containerID="35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53"
Sep 15 12:28:38.689278 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:38.689256    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53\": container with ID starting with 35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53 not found: ID does not exist" containerID="35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53"
Sep 15 12:28:38.689320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.689289    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53"} err="failed to get container status \"35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53\": rpc error: code = NotFound desc = could not find container \"35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53\": container with ID starting with 35d747822996b945b1ab3cd389c43b07fe2b721de6c70b02a1a08f989c089c53 not found: ID does not exist"
Sep 15 12:28:38.689320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.689309    2583 scope.go:117] "RemoveContainer" containerID="e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"
Sep 15 12:28:38.689586 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:38.689566    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40\": container with ID starting with e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40 not found: ID does not exist" containerID="e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"
Sep 15 12:28:38.689633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.689591    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40"} err="failed to get container status \"e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40\": rpc error: code = NotFound desc = could not find container \"e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40\": container with ID starting with e5509ea0bbcb059df1b07d7a724b1126df53f729d9a8255896a85b7849fe4c40 not found: ID does not exist"
Sep 15 12:28:38.735174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.735146    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.735174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.735177    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.735337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.735190    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9l2\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-kube-api-access-xr9l2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.735337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.735204    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.735337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.735217    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b54dcf7-1f29-49bd-bfd5-2650deeccda7-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:38.736198 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.736179    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"]
Sep 15 12:28:38.754751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.754726    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b0fd-openshift-default-67576b7b9b-xxq2s"]
Sep 15 12:28:38.825385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:38.825352    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:39.074129 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:39.074095    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:39.806937 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:39.806899    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" path="/var/lib/kubelet/pods/3b54dcf7-1f29-49bd-bfd5-2650deeccda7/volumes"
Sep 15 12:28:40.112673 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:40.112637    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:44.546999 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.546962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"]
Sep 15 12:28:44.547491 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.547225    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" podUID="d2d79c4b-25bb-4f4b-b6b5-697e9331236a" containerName="maas-api" containerID="cri-o://11b04ad1e2e4d55b63c11ab527f9ef09afc484691138df12f14992da366f4da2" gracePeriod=30
Sep 15 12:28:44.699981 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.699949    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2d79c4b-25bb-4f4b-b6b5-697e9331236a" containerID="11b04ad1e2e4d55b63c11ab527f9ef09afc484691138df12f14992da366f4da2" exitCode=0
Sep 15 12:28:44.700135 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.700047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" event={"ID":"d2d79c4b-25bb-4f4b-b6b5-697e9331236a","Type":"ContainerDied","Data":"11b04ad1e2e4d55b63c11ab527f9ef09afc484691138df12f14992da366f4da2"}
Sep 15 12:28:44.798013 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.797950    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:28:44.859476 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.859433    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:44.894334 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.894302    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls\") pod \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") "
Sep 15 12:28:44.894521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.894345    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zp9m\" (UniqueName: \"kubernetes.io/projected/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-kube-api-access-9zp9m\") pod \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") "
Sep 15 12:28:44.894521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.894409    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-metrics-certs\") pod \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") "
Sep 15 12:28:44.894521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.894446    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-openshift-service-ca\") pod \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\" (UID: \"d2d79c4b-25bb-4f4b-b6b5-697e9331236a\") "
Sep 15 12:28:44.894869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.894841    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2d79c4b-25bb-4f4b-b6b5-697e9331236a" (UID: "d2d79c4b-25bb-4f4b-b6b5-697e9331236a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:44.897079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.896902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-kube-api-access-9zp9m" (OuterVolumeSpecName: "kube-api-access-9zp9m") pod "d2d79c4b-25bb-4f4b-b6b5-697e9331236a" (UID: "d2d79c4b-25bb-4f4b-b6b5-697e9331236a"). InnerVolumeSpecName "kube-api-access-9zp9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:44.897079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.897001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2d79c4b-25bb-4f4b-b6b5-697e9331236a" (UID: "d2d79c4b-25bb-4f4b-b6b5-697e9331236a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:44.897079 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.897012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2d79c4b-25bb-4f4b-b6b5-697e9331236a" (UID: "d2d79c4b-25bb-4f4b-b6b5-697e9331236a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:44.961064 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.960999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:44.995509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.995479    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:44.995509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.995507    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:44.995509 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.995517    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zp9m\" (UniqueName: \"kubernetes.io/projected/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-kube-api-access-9zp9m\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:44.995758 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:44.995526    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d79c4b-25bb-4f4b-b6b5-697e9331236a-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.129171 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.129125    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"]
Sep 15 12:28:45.129443 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.129407    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" podUID="db24a8af-e6f2-4246-8654-267122705fb6" containerName="maas-api" containerID="cri-o://5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4" gracePeriod=30
Sep 15 12:28:45.166035 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.165975    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:45.242622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.242586    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:45.376703 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.376674    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:45.377573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.377552    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:28:45.501651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.501559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-metrics-certs\") pod \"db24a8af-e6f2-4246-8654-267122705fb6\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") "
Sep 15 12:28:45.501651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.501603    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfpz4\" (UniqueName: \"kubernetes.io/projected/db24a8af-e6f2-4246-8654-267122705fb6-kube-api-access-pfpz4\") pod \"db24a8af-e6f2-4246-8654-267122705fb6\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") "
Sep 15 12:28:45.501881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.501670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls\") pod \"db24a8af-e6f2-4246-8654-267122705fb6\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") "
Sep 15 12:28:45.501881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.501712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db24a8af-e6f2-4246-8654-267122705fb6-openshift-service-ca\") pod \"db24a8af-e6f2-4246-8654-267122705fb6\" (UID: \"db24a8af-e6f2-4246-8654-267122705fb6\") "
Sep 15 12:28:45.502164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.502128    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db24a8af-e6f2-4246-8654-267122705fb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db24a8af-e6f2-4246-8654-267122705fb6" (UID: "db24a8af-e6f2-4246-8654-267122705fb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:45.503752 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.503722    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db24a8af-e6f2-4246-8654-267122705fb6" (UID: "db24a8af-e6f2-4246-8654-267122705fb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:45.503861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.503766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "db24a8af-e6f2-4246-8654-267122705fb6" (UID: "db24a8af-e6f2-4246-8654-267122705fb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:28:45.503861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.503799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db24a8af-e6f2-4246-8654-267122705fb6-kube-api-access-pfpz4" (OuterVolumeSpecName: "kube-api-access-pfpz4") pod "db24a8af-e6f2-4246-8654-267122705fb6" (UID: "db24a8af-e6f2-4246-8654-267122705fb6"). InnerVolumeSpecName "kube-api-access-pfpz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:45.521847 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.521804    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:45.603376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.603323    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.603376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.603360    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfpz4\" (UniqueName: \"kubernetes.io/projected/db24a8af-e6f2-4246-8654-267122705fb6-kube-api-access-pfpz4\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.603376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.603376    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db24a8af-e6f2-4246-8654-267122705fb6-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.603972 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.603391    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db24a8af-e6f2-4246-8654-267122705fb6-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:45.654205 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.654161    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:45.707379 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.707346    2583 generic.go:356] "Generic (PLEG): container finished" podID="db24a8af-e6f2-4246-8654-267122705fb6" containerID="5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4" exitCode=0
Sep 15 12:28:45.707593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.707411    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"
Sep 15 12:28:45.707593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.707523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" event={"ID":"db24a8af-e6f2-4246-8654-267122705fb6","Type":"ContainerDied","Data":"5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4"}
Sep 15 12:28:45.707593 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.707578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4" event={"ID":"db24a8af-e6f2-4246-8654-267122705fb6","Type":"ContainerDied","Data":"763a0fb4752188b9f45a3d48498d0b47449d26321a6f1aece7de89583d5482f8"}
Sep 15 12:28:45.707773 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.707602    2583 scope.go:117] "RemoveContainer" containerID="5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4"
Sep 15 12:28:45.709863 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.709798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv" event={"ID":"d2d79c4b-25bb-4f4b-b6b5-697e9331236a","Type":"ContainerDied","Data":"4b612e9cb0aade459d4fb4780b8802d2cd92333dc37a7f233f24b54b8d7e300f"}
Sep 15 12:28:45.709944 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.709900    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"
Sep 15 12:28:45.720466 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.720444    2583 scope.go:117] "RemoveContainer" containerID="5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4"
Sep 15 12:28:45.720860 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:45.720834    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4\": container with ID starting with 5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4 not found: ID does not exist" containerID="5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4"
Sep 15 12:28:45.720950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.720872    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4"} err="failed to get container status \"5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4\": rpc error: code = NotFound desc = could not find container \"5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4\": container with ID starting with 5ec274e260353e97bd01d11046df7d73f697ad50e7a91434375646412edb91f4 not found: ID does not exist"
Sep 15 12:28:45.720950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.720898    2583 scope.go:117] "RemoveContainer" containerID="11b04ad1e2e4d55b63c11ab527f9ef09afc484691138df12f14992da366f4da2"
Sep 15 12:28:45.754691 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.754617    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"]
Sep 15 12:28:45.757326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.757301    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba366c-646bdf67f5-9fxgv"]
Sep 15 12:28:45.789009 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.788975    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"]
Sep 15 12:28:45.796492 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.796465    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9ba48f-589d4586b4-dprd4"]
Sep 15 12:28:45.807757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.807728    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d79c4b-25bb-4f4b-b6b5-697e9331236a" path="/var/lib/kubelet/pods/d2d79c4b-25bb-4f4b-b6b5-697e9331236a/volumes"
Sep 15 12:28:45.808388 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:45.808365    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db24a8af-e6f2-4246-8654-267122705fb6" path="/var/lib/kubelet/pods/db24a8af-e6f2-4246-8654-267122705fb6/volumes"
Sep 15 12:28:46.014656 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:46.014557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:46.828861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:46.828820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:48.068555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:48.068521    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:49.205861 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:49.205819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:54.369087 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:54.368996    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"]
Sep 15 12:28:54.369499 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:54.369241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" containerID="cri-o://51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2" gracePeriod=30
Sep 15 12:28:55.360881 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:55.360843    2583 patch_prober.go:28] interesting pod/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:28:55.361111 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:55.360907    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:28:56.093705 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:56.093664    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:56.196814 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:56.196784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:59.621244 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.621217    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:28:59.637302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-data\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7202520-11ad-406a-b2a3-42c493494693-istio-podinfo\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637304    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-credential-socket\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-istio-token\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637399    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-envoy\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637446    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vtm2\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-kube-api-access-7vtm2\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637487    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:59.637790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637517    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-certs\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-socket\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637593    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7202520-11ad-406a-b2a3-42c493494693-istiod-ca-cert\") pod \"a7202520-11ad-406a-b2a3-42c493494693\" (UID: \"a7202520-11ad-406a-b2a3-42c493494693\") "
Sep 15 12:28:59.637790 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637640    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:59.638018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637841    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:59.638018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637935    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.638018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637955    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.638018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.637970    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.638355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.638206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:59.638355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.638244    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7202520-11ad-406a-b2a3-42c493494693-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:28:59.639624 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.639597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7202520-11ad-406a-b2a3-42c493494693-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:28:59.639784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.639764    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-kube-api-access-7vtm2" (OuterVolumeSpecName: "kube-api-access-7vtm2") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "kube-api-access-7vtm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:59.639928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.639908    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:28:59.639928 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.639920    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7202520-11ad-406a-b2a3-42c493494693" (UID: "a7202520-11ad-406a-b2a3-42c493494693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:28:59.738882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.738846    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.738882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.738878    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7202520-11ad-406a-b2a3-42c493494693-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.738882 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.738888    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7202520-11ad-406a-b2a3-42c493494693-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.739147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.738897    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.739147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.738905    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7202520-11ad-406a-b2a3-42c493494693-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.739147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.738914    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vtm2\" (UniqueName: \"kubernetes.io/projected/a7202520-11ad-406a-b2a3-42c493494693-kube-api-access-7vtm2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:28:59.792603 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.792569    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7202520-11ad-406a-b2a3-42c493494693" containerID="51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2" exitCode=0
Sep 15 12:28:59.792784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.792630    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"
Sep 15 12:28:59.792784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.792666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" event={"ID":"a7202520-11ad-406a-b2a3-42c493494693","Type":"ContainerDied","Data":"51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2"}
Sep 15 12:28:59.792784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.792732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68" event={"ID":"a7202520-11ad-406a-b2a3-42c493494693","Type":"ContainerDied","Data":"3d199247d48242473839e937e7731c7c20cad149d85e798c8658ec7e3d3b1044"}
Sep 15 12:28:59.792784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.792764    2583 scope.go:117] "RemoveContainer" containerID="51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2"
Sep 15 12:28:59.803862 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.803840    2583 scope.go:117] "RemoveContainer" containerID="a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"
Sep 15 12:28:59.817648 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.817579    2583 scope.go:117] "RemoveContainer" containerID="51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2"
Sep 15 12:28:59.818832 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:59.818804    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2\": container with ID starting with 51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2 not found: ID does not exist" containerID="51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2"
Sep 15 12:28:59.818960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.818846    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2"} err="failed to get container status \"51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2\": rpc error: code = NotFound desc = could not find container \"51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2\": container with ID starting with 51e8a1c3d1e053889988adbad58362c800d1330443335bc876df17a57febbbd2 not found: ID does not exist"
Sep 15 12:28:59.818960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.818873    2583 scope.go:117] "RemoveContainer" containerID="a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"
Sep 15 12:28:59.819259 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:28:59.819229    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4\": container with ID starting with a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4 not found: ID does not exist" containerID="a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"
Sep 15 12:28:59.819342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.819269    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4"} err="failed to get container status \"a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4\": rpc error: code = NotFound desc = could not find container \"a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4\": container with ID starting with a8407c66c824ef61b1280fd367e05a73e4933666b26364ec74abac68992056f4 not found: ID does not exist"
Sep 15 12:28:59.820849 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.820828    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"]
Sep 15 12:28:59.824745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:28:59.824724    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9ba48f-openshift-default-8f8d7c6d8-shb68"]
Sep 15 12:29:01.807337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:01.807303    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7202520-11ad-406a-b2a3-42c493494693" path="/var/lib/kubelet/pods/a7202520-11ad-406a-b2a3-42c493494693/volumes"
Sep 15 12:29:07.538117 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:07.538080    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:10.179513 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:10.179470    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:13.710544 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:13.710507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:13.812052 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:13.811998    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:13.912610 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:13.912562    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:14.419370 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:14.419337    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:20.131250 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:20.131208    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:30.846683 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:30.846647    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:40.816243 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:40.816207    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"]
Sep 15 12:29:42.424764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:42.424729    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:42.485341 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:42.485304    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:48.460324 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:48.460288    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:55.837149 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:55.837100    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j" podUID="4c742e5c-9c17-4660-9688-f53b3690566d" containerName="main" containerID="cri-o://9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180" gracePeriod=45
Sep 15 12:29:56.106682 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.106588    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:29:56.109539 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.109508    2583 generic.go:356] "Generic (PLEG): container finished" podID="4c742e5c-9c17-4660-9688-f53b3690566d" containerID="9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180" exitCode=0
Sep 15 12:29:56.109675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.109550    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"
Sep 15 12:29:56.109675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.109591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j" event={"ID":"4c742e5c-9c17-4660-9688-f53b3690566d","Type":"ContainerDied","Data":"9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180"}
Sep 15 12:29:56.109675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.109633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j" event={"ID":"4c742e5c-9c17-4660-9688-f53b3690566d","Type":"ContainerDied","Data":"3efa379d99802cca37b7ece98cace15fc4e8f6302696f94ee457c2a224da63d7"}
Sep 15 12:29:56.109675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.109650    2583 scope.go:117] "RemoveContainer" containerID="9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180"
Sep 15 12:29:56.119002 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.118974    2583 scope.go:117] "RemoveContainer" containerID="9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180"
Sep 15 12:29:56.119352 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:29:56.119327    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180\": container with ID starting with 9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180 not found: ID does not exist" containerID="9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180"
Sep 15 12:29:56.119428 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.119366    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180"} err="failed to get container status \"9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180\": rpc error: code = NotFound desc = could not find container \"9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180\": container with ID starting with 9f9af312bbc7ad2e7892ea912c317563719005e9bb69a19a6709e13a8e82a180 not found: ID does not exist"
Sep 15 12:29:56.166301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bqnh\" (UniqueName: \"kubernetes.io/projected/4c742e5c-9c17-4660-9688-f53b3690566d-kube-api-access-2bqnh\") pod \"4c742e5c-9c17-4660-9688-f53b3690566d\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") "
Sep 15 12:29:56.166301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166304    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-tmp-dir\") pod \"4c742e5c-9c17-4660-9688-f53b3690566d\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") "
Sep 15 12:29:56.166554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166358    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c742e5c-9c17-4660-9688-f53b3690566d-tls-certs\") pod \"4c742e5c-9c17-4660-9688-f53b3690566d\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") "
Sep 15 12:29:56.166554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-model-cache\") pod \"4c742e5c-9c17-4660-9688-f53b3690566d\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") "
Sep 15 12:29:56.166554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-home\") pod \"4c742e5c-9c17-4660-9688-f53b3690566d\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") "
Sep 15 12:29:56.166554 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166448    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-dshm\") pod \"4c742e5c-9c17-4660-9688-f53b3690566d\" (UID: \"4c742e5c-9c17-4660-9688-f53b3690566d\") "
Sep 15 12:29:56.166767 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c742e5c-9c17-4660-9688-f53b3690566d" (UID: "4c742e5c-9c17-4660-9688-f53b3690566d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:56.166767 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166678    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-home" (OuterVolumeSpecName: "home") pod "4c742e5c-9c17-4660-9688-f53b3690566d" (UID: "4c742e5c-9c17-4660-9688-f53b3690566d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:56.166767 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c742e5c-9c17-4660-9688-f53b3690566d" (UID: "4c742e5c-9c17-4660-9688-f53b3690566d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:56.166877 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166828    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-tmp-dir\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:29:56.166877 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166844    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-model-cache\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:29:56.166877 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.166852    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-home\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:29:56.168584 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.168556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-dshm" (OuterVolumeSpecName: "dshm") pod "4c742e5c-9c17-4660-9688-f53b3690566d" (UID: "4c742e5c-9c17-4660-9688-f53b3690566d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:56.168699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.168657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c742e5c-9c17-4660-9688-f53b3690566d-kube-api-access-2bqnh" (OuterVolumeSpecName: "kube-api-access-2bqnh") pod "4c742e5c-9c17-4660-9688-f53b3690566d" (UID: "4c742e5c-9c17-4660-9688-f53b3690566d"). InnerVolumeSpecName "kube-api-access-2bqnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:29:56.168699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.168686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c742e5c-9c17-4660-9688-f53b3690566d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c742e5c-9c17-4660-9688-f53b3690566d" (UID: "4c742e5c-9c17-4660-9688-f53b3690566d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:29:56.267672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.267633    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c742e5c-9c17-4660-9688-f53b3690566d-dshm\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:29:56.267672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.267666    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bqnh\" (UniqueName: \"kubernetes.io/projected/4c742e5c-9c17-4660-9688-f53b3690566d-kube-api-access-2bqnh\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:29:56.267672 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.267680    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c742e5c-9c17-4660-9688-f53b3690566d-tls-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:29:56.431147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.431107    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"]
Sep 15 12:29:56.433633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:56.433609    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-969130/auth-test-model-969130-kserve-f56db8cbc-lzc5j"]
Sep 15 12:29:57.806690 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:29:57.806645    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c742e5c-9c17-4660-9688-f53b3690566d" path="/var/lib/kubelet/pods/4c742e5c-9c17-4660-9688-f53b3690566d/volumes"
Sep 15 12:30:02.112860 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:30:02.112824    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:11.991793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:30:11.991732    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:51.099227 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:30:51.099183    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:51.966268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:30:51.966232    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:58.031487 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:30:58.031454    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:35.231215 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:31:35.231175    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:59.892827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:31:59.892797    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:31:59.895553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:31:59.895526    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:32:12.304546 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:12.304500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:13.207551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:13.207514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:13.698039 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:13.697999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:14.016009 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.015932    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"]
Sep 15 12:32:14.016388 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016372    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016390    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016399    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016404    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016417    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a5da10-3ddd-4880-8d47-001c454ca14a" containerName="maas-api"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016423    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a5da10-3ddd-4880-8d47-001c454ca14a" containerName="maas-api"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016431    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016436    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy"
Sep 15 12:32:14.016441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016441    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016448    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016454    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016459    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016465    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b72742-04c7-41c4-8496-d05861a7e838" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016469    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b72742-04c7-41c4-8496-d05861a7e838" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016479    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c742e5c-9c17-4660-9688-f53b3690566d" containerName="main"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016484    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c742e5c-9c17-4660-9688-f53b3690566d" containerName="main"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016489    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d79c4b-25bb-4f4b-b6b5-697e9331236a" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016494    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d79c4b-25bb-4f4b-b6b5-697e9331236a" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016500    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32270fc2-1c0e-4f11-8741-bd4877bb0283" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016504    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="32270fc2-1c0e-4f11-8741-bd4877bb0283" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016513    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016518    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016526    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016531    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016541    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db24a8af-e6f2-4246-8654-267122705fb6" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016546    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="db24a8af-e6f2-4246-8654-267122705fb6" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016550    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb3fea2-df4e-472f-9f34-11ec2a62bbfa" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016555    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb3fea2-df4e-472f-9f34-11ec2a62bbfa" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016615    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d79c4b-25bb-4f4b-b6b5-697e9331236a" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016624    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a5da10-3ddd-4880-8d47-001c454ca14a" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016632    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b72742-04c7-41c4-8496-d05861a7e838" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016638    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="32270fc2-1c0e-4f11-8741-bd4877bb0283" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016644    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016651    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb3fea2-df4e-472f-9f34-11ec2a62bbfa" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016656    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c742e5c-9c17-4660-9688-f53b3690566d" containerName="main"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016661    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016666    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016671    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7202520-11ad-406a-b2a3-42c493494693" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016677    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016682    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="db24a8af-e6f2-4246-8654-267122705fb6" containerName="maas-api"
Sep 15 12:32:14.016692 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.016687    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy"
Sep 15 12:32:14.019835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.019818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.022534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.022513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ed21e2-openshift-default-dockercfg-splx2\""
Sep 15 12:32:14.036069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.036043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"]
Sep 15 12:32:14.072577 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-data\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-certs\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-envoy\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-token\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-socket\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-credential-socket\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27dea5d-1da6-4b11-8f1b-278e95154db6-istiod-ca-cert\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djgs\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-kube-api-access-4djgs\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.072983 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.072967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-podinfo\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.126537 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.126502    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:14.173797 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.173764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-credential-socket\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.173985 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.173802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27dea5d-1da6-4b11-8f1b-278e95154db6-istiod-ca-cert\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.173985 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.173935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djgs\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-kube-api-access-4djgs\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174128 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.173987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-podinfo\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174128 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-data\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174128 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-certs\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174128 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-envoy\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-token\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-socket\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174365 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-credential-socket\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-certs\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27dea5d-1da6-4b11-8f1b-278e95154db6-istiod-ca-cert\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174626 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-socket\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.174664 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.174647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-data\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.176460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.176439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-envoy\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.176843 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.176828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-podinfo\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.181699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.181670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-token\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.181819 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.181774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djgs\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-kube-api-access-4djgs\") pod \"e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") " pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.330770 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.330727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:14.352578 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.352531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:14.612322 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.612285    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"]
Sep 15 12:32:14.615098 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:32:14.615065    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27dea5d_1da6_4b11_8f1b_278e95154db6.slice/crio-40d1f1a86060e253a5e701d7e4ac986b3a9ae524da75b7f7485f404ec040903b WatchSource:0}: Error finding container 40d1f1a86060e253a5e701d7e4ac986b3a9ae524da75b7f7485f404ec040903b: Status 404 returned error can't find the container with id 40d1f1a86060e253a5e701d7e4ac986b3a9ae524da75b7f7485f404ec040903b
Sep 15 12:32:14.617844 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.617804    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:14.617917 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.617898    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:14.617966 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.617948    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:14.849442 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.849398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" event={"ID":"e27dea5d-1da6-4b11-8f1b-278e95154db6","Type":"ContainerStarted","Data":"55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"}
Sep 15 12:32:14.849885 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.849457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" event={"ID":"e27dea5d-1da6-4b11-8f1b-278e95154db6","Type":"ContainerStarted","Data":"40d1f1a86060e253a5e701d7e4ac986b3a9ae524da75b7f7485f404ec040903b"}
Sep 15 12:32:14.868597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:14.868488    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podStartSLOduration=1.868465966 podStartE2EDuration="1.868465966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:32:14.867145985 +0000 UTC m=+1815.563705581" watchObservedRunningTime="2026-09-15 12:32:14.868465966 +0000 UTC m=+1815.565025544"
Sep 15 12:32:15.330879 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:15.330850    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:15.336404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:15.336379    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:15.855643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:15.855612    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:15.856643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:15.856622    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:16.593039 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:16.592997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:16.879409 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:16.879330    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:16.978721 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:16.978684    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:17.610430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:17.610392    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:18.276346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.276305    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"]
Sep 15 12:32:18.277004 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.276981    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy"
Sep 15 12:32:18.277131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.277007    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="da032b7b-cb6e-4c88-862e-9e7fde782f65" containerName="istio-proxy"
Sep 15 12:32:18.277199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.277169    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2417dd-b3f7-4e42-9012-7c99cc34ab71" containerName="istio-proxy"
Sep 15 12:32:18.277199 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.277191    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b54dcf7-1f29-49bd-bfd5-2650deeccda7" containerName="istio-proxy"
Sep 15 12:32:18.281460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.281438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.283622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.283594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ed21e2-serving-cert\""
Sep 15 12:32:18.291615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.291544    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"]
Sep 15 12:32:18.325467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.325438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:18.416227 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.416187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2llv\" (UniqueName: \"kubernetes.io/projected/3af07cef-d1fc-41cd-8fac-5855008c7fff-kube-api-access-r2llv\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.416433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.416240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.416433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.416384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.416433 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.416413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3af07cef-d1fc-41cd-8fac-5855008c7fff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.424219 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.424182    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:18.517841 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.517803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.518049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.517846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3af07cef-d1fc-41cd-8fac-5855008c7fff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.518049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.517893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2llv\" (UniqueName: \"kubernetes.io/projected/3af07cef-d1fc-41cd-8fac-5855008c7fff-kube-api-access-r2llv\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.518049 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.517917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.518641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.518614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3af07cef-d1fc-41cd-8fac-5855008c7fff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.520604 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.520581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-metrics-certs\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.520678 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.520581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.528615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.528541    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:18.528865 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.528828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2llv\" (UniqueName: \"kubernetes.io/projected/3af07cef-d1fc-41cd-8fac-5855008c7fff-kube-api-access-r2llv\") pod \"maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.596093 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.596036    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:18.776667 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.776633    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:18.874973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.874946    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"]
Sep 15 12:32:18.876798 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.876769    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:18.877140 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:32:18.877114    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af07cef_d1fc_41cd_8fac_5855008c7fff.slice/crio-9b140f48e9bc4142f14000f44a7f2467f07cd25ddec084fa58299e7fe41c2a4d WatchSource:0}: Error finding container 9b140f48e9bc4142f14000f44a7f2467f07cd25ddec084fa58299e7fe41c2a4d: Status 404 returned error can't find the container with id 9b140f48e9bc4142f14000f44a7f2467f07cd25ddec084fa58299e7fe41c2a4d
Sep 15 12:32:18.878516 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:18.878498    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:32:19.878699 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:19.878667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz" event={"ID":"3af07cef-d1fc-41cd-8fac-5855008c7fff","Type":"ContainerStarted","Data":"e39a44ad7ca2fbcfaaabfa06c65e3a9656f7811ef2be44a232a6175dfd4d9b9b"}
Sep 15 12:32:19.879131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:19.878720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz" event={"ID":"3af07cef-d1fc-41cd-8fac-5855008c7fff","Type":"ContainerStarted","Data":"9b140f48e9bc4142f14000f44a7f2467f07cd25ddec084fa58299e7fe41c2a4d"}
Sep 15 12:32:19.879131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:19.878770    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:19.896038 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:19.895981    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz" podStartSLOduration=1.456393917 podStartE2EDuration="1.895963543s" totalImagesPullingTime="439.569626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:32:19.894439741 +0000 UTC m=+1820.590999337" watchObservedRunningTime="2026-09-15 12:32:19.895963543 +0000 UTC m=+1820.592523121"
Sep 15 12:32:25.890438 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:25.890403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:32:31.464203 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.464165    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:47904->10.132.0.85:15021: read: connection reset by peer" start-of-body=
Sep 15 12:32:31.464579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.464242    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:47904->10.132.0.85:15021: read: connection reset by peer"
Sep 15 12:32:31.953592 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.953554    2583 generic.go:356] "Generic (PLEG): container finished" podID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerID="55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade" exitCode=0
Sep 15 12:32:31.953778 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.953628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" event={"ID":"e27dea5d-1da6-4b11-8f1b-278e95154db6","Type":"ContainerDied","Data":"55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"}
Sep 15 12:32:31.954083 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.954060    2583 scope.go:117] "RemoveContainer" containerID="55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"
Sep 15 12:32:31.955213 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.955185    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:31.955275 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.955262    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:31.955317 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:31.955294    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 12:32:32.964207 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:32.964171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" event={"ID":"e27dea5d-1da6-4b11-8f1b-278e95154db6","Type":"ContainerStarted","Data":"5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8"}
Sep 15 12:32:33.331835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:33.331796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:33.332172 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:33.331852    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 12:32:33.332316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:33.332206    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 12:32:34.331967 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:34.331911    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 12:32:34.332516 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:34.331986    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 12:32:35.331869 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:35.331826    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 12:32:35.332059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:35.331884    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 12:32:36.331807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:36.331765    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 12:32:36.332078 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:36.331847    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 12:32:37.331741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:37.331699    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 12:32:37.331963 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:37.331772    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 12:32:38.336164 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:38.336123    2583 patch_prober.go:28] interesting pod/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Sep 15 12:32:38.336796 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:38.336758    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Sep 15 12:32:39.335338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:39.335306    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:39.335730 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:39.335702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:39.336354 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:39.336333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:32:39.377686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:39.377648    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:43.046302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:43.046263    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:43.317920 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:43.317878    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:45.407328 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:45.407285    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:45.739609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:45.739510    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:46.056546 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:46.056462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:46.150258 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:46.150218    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:46.250460 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:46.250419    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:46.991293 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:46.991261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:47.771536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:47.771494    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:48.593138 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.593096    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"]
Sep 15 12:32:48.595876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.595855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.598202 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.598177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7b204e-serving-cert\""
Sep 15 12:32:48.608828 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.608804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"]
Sep 15 12:32:48.628590 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.628554    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:48.643804 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.643780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6z42\" (UniqueName: \"kubernetes.io/projected/d01a2111-500b-4e67-ba7e-75300b02e040-kube-api-access-l6z42\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.643942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.643834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-metrics-certs\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.643942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.643866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01a2111-500b-4e67-ba7e-75300b02e040-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.643942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.643915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.723316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.723283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:48.745523 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.745490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.745710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.745560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6z42\" (UniqueName: \"kubernetes.io/projected/d01a2111-500b-4e67-ba7e-75300b02e040-kube-api-access-l6z42\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.745710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.745591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-metrics-certs\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.745710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.745610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01a2111-500b-4e67-ba7e-75300b02e040-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.746320 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.746296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01a2111-500b-4e67-ba7e-75300b02e040-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.748008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.747986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.748106 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.748049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-metrics-certs\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.756997 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.756977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6z42\" (UniqueName: \"kubernetes.io/projected/d01a2111-500b-4e67-ba7e-75300b02e040-kube-api-access-l6z42\") pod \"maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:48.809742 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.809705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:48.910280 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:48.910191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:49.187011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:49.186958    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"]
Sep 15 12:32:49.189683 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:32:49.189653    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01a2111_500b_4e67_ba7e_75300b02e040.slice/crio-06f76b9cd30cabeeb13bda3cb94803cd5e38bf026c67f86ed0ccde0c5c5038a5 WatchSource:0}: Error finding container 06f76b9cd30cabeeb13bda3cb94803cd5e38bf026c67f86ed0ccde0c5c5038a5: Status 404 returned error can't find the container with id 06f76b9cd30cabeeb13bda3cb94803cd5e38bf026c67f86ed0ccde0c5c5038a5
Sep 15 12:32:49.192805 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:49.192777    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:49.284441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:49.284406    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:50.062467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:50.062431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr" event={"ID":"d01a2111-500b-4e67-ba7e-75300b02e040","Type":"ContainerStarted","Data":"201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de"}
Sep 15 12:32:50.062850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:50.062490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr" event={"ID":"d01a2111-500b-4e67-ba7e-75300b02e040","Type":"ContainerStarted","Data":"06f76b9cd30cabeeb13bda3cb94803cd5e38bf026c67f86ed0ccde0c5c5038a5"}
Sep 15 12:32:50.062850 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:50.062513    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:50.080558 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:50.080505    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr" podStartSLOduration=1.748396667 podStartE2EDuration="2.080489473s" totalImagesPullingTime="332.092806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:32:50.079268189 +0000 UTC m=+1850.775827815" watchObservedRunningTime="2026-09-15 12:32:50.080489473 +0000 UTC m=+1850.777049051"
Sep 15 12:32:56.073171 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:56.073140    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:32:59.285521 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:32:59.285477    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:09.691551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:09.691511    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:10.893726 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:10.893693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:11.038287 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:11.038248    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:11.700062 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:11.700005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:12.382815 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:12.382777    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:13.160750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:13.160717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:13.900563 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:13.900513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:20.158220 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:20.158177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:25.526785 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:25.526744    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:49.106457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:49.106422    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:57.899159 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:57.899116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:58.210114 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:58.209990    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:59.351636 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:33:59.351590    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:00.999586 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:00.999544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:01.649296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:01.649262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:01.748011 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:01.747925    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:02.794236 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:02.794185    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:03.592966 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:03.592931    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:04.385617 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:04.385578    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:05.086383 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.086344    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"]
Sep 15 12:34:05.089710 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.089676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.092069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.092040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-9725ba-serving-cert\""
Sep 15 12:34:05.099836 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.099805    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"]
Sep 15 12:34:05.124174 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.124136    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:05.178536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.178500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnb87\" (UniqueName: \"kubernetes.io/projected/2f468c53-9680-43fd-ab80-e6bd9c7761bb-kube-api-access-qnb87\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.178536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.178548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.178812 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.178571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f468c53-9680-43fd-ab80-e6bd9c7761bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.178812 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.178687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.221183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.221146    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:05.280196 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.280165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnb87\" (UniqueName: \"kubernetes.io/projected/2f468c53-9680-43fd-ab80-e6bd9c7761bb-kube-api-access-qnb87\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.280423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.280212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.280423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.280250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f468c53-9680-43fd-ab80-e6bd9c7761bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.280529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.280418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.280937 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.280915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f468c53-9680-43fd-ab80-e6bd9c7761bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.282930 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.282902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.283059 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.282980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.288651 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.288629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnb87\" (UniqueName: \"kubernetes.io/projected/2f468c53-9680-43fd-ab80-e6bd9c7761bb-kube-api-access-qnb87\") pod \"maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.403948 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.403856    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:05.578746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.578710    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:05.682643 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.682610    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"]
Sep 15 12:34:05.684817 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:34:05.684787    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f468c53_9680_43fd_ab80_e6bd9c7761bb.slice/crio-7ee6217b1b2dac82d0d58d37dfe28ef9fd04e13e1eea994c17f73a590a0ed49b WatchSource:0}: Error finding container 7ee6217b1b2dac82d0d58d37dfe28ef9fd04e13e1eea994c17f73a590a0ed49b: Status 404 returned error can't find the container with id 7ee6217b1b2dac82d0d58d37dfe28ef9fd04e13e1eea994c17f73a590a0ed49b
Sep 15 12:34:05.971293 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:05.971189    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:06.476292 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:06.476256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c" event={"ID":"2f468c53-9680-43fd-ab80-e6bd9c7761bb","Type":"ContainerStarted","Data":"1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8"}
Sep 15 12:34:06.476725 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:06.476314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c" event={"ID":"2f468c53-9680-43fd-ab80-e6bd9c7761bb","Type":"ContainerStarted","Data":"7ee6217b1b2dac82d0d58d37dfe28ef9fd04e13e1eea994c17f73a590a0ed49b"}
Sep 15 12:34:06.476725 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:06.476487    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:06.494990 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:06.494895    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c" podStartSLOduration=0.97674418 podStartE2EDuration="1.49487337s" totalImagesPullingTime="518.12919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:34:06.493730991 +0000 UTC m=+1927.190290609" watchObservedRunningTime="2026-09-15 12:34:06.49487337 +0000 UTC m=+1927.191432948"
Sep 15 12:34:12.487434 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:12.487401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:28.059121 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:28.059075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:28.904515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:28.904473    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:29.598749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:29.598714    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:30.230233 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:30.230175    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:30.986207 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:30.986162    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:32.256572 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:32.256529    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:32.356224 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:32.356184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:33.499416 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:33.499374    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:34.187621 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:34.187573    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:35.088420 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:35.088383    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:35.799446 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:35.799413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:36.462454 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:36.462406    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:37.173537 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:37.173500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:43.292641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.292597    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"]
Sep 15 12:34:43.293597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.293532    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c" podUID="2f468c53-9680-43fd-ab80-e6bd9c7761bb" containerName="maas-api" containerID="cri-o://1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8" gracePeriod=30
Sep 15 12:34:43.324316 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.324284    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:43.422434 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.422399    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:43.516807 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.516755    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:43.554898 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.554838    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:43.580633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.580594    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:43.675965 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.675930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnb87\" (UniqueName: \"kubernetes.io/projected/2f468c53-9680-43fd-ab80-e6bd9c7761bb-kube-api-access-qnb87\") pod \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") "
Sep 15 12:34:43.676161 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.676048    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-metrics-certs\") pod \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") "
Sep 15 12:34:43.676161 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.676083    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-maas-api-tls\") pod \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") "
Sep 15 12:34:43.676261 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.676163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f468c53-9680-43fd-ab80-e6bd9c7761bb-openshift-service-ca\") pod \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\" (UID: \"2f468c53-9680-43fd-ab80-e6bd9c7761bb\") "
Sep 15 12:34:43.676585 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.676543    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f468c53-9680-43fd-ab80-e6bd9c7761bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f468c53-9680-43fd-ab80-e6bd9c7761bb" (UID: "2f468c53-9680-43fd-ab80-e6bd9c7761bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:34:43.678152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.678129    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f468c53-9680-43fd-ab80-e6bd9c7761bb" (UID: "2f468c53-9680-43fd-ab80-e6bd9c7761bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:34:43.678264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.678162    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f468c53-9680-43fd-ab80-e6bd9c7761bb" (UID: "2f468c53-9680-43fd-ab80-e6bd9c7761bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:34:43.678264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.678177    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f468c53-9680-43fd-ab80-e6bd9c7761bb-kube-api-access-qnb87" (OuterVolumeSpecName: "kube-api-access-qnb87") pod "2f468c53-9680-43fd-ab80-e6bd9c7761bb" (UID: "2f468c53-9680-43fd-ab80-e6bd9c7761bb"). InnerVolumeSpecName "kube-api-access-qnb87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:34:43.692794 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.692759    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f468c53-9680-43fd-ab80-e6bd9c7761bb" containerID="1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8" exitCode=0
Sep 15 12:34:43.692936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.692820    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"
Sep 15 12:34:43.692936 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.692826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c" event={"ID":"2f468c53-9680-43fd-ab80-e6bd9c7761bb","Type":"ContainerDied","Data":"1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8"}
Sep 15 12:34:43.693091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.692957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c" event={"ID":"2f468c53-9680-43fd-ab80-e6bd9c7761bb","Type":"ContainerDied","Data":"7ee6217b1b2dac82d0d58d37dfe28ef9fd04e13e1eea994c17f73a590a0ed49b"}
Sep 15 12:34:43.693091 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.692990    2583 scope.go:117] "RemoveContainer" containerID="1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8"
Sep 15 12:34:43.707350 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.707324    2583 scope.go:117] "RemoveContainer" containerID="1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8"
Sep 15 12:34:43.708859 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:34:43.708823    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8\": container with ID starting with 1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8 not found: ID does not exist" containerID="1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8"
Sep 15 12:34:43.709008 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.708867    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8"} err="failed to get container status \"1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8\": rpc error: code = NotFound desc = could not find container \"1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8\": container with ID starting with 1ed470b226b913bfa8f2b995515c1b43fb921d0a0c9bdf4ab2adcace03400bf8 not found: ID does not exist"
Sep 15 12:34:43.722065 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.722003    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"]
Sep 15 12:34:43.724052 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.724032    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9725ba-57dc45557b-nt58c"]
Sep 15 12:34:43.777100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.777067    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f468c53-9680-43fd-ab80-e6bd9c7761bb-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:34:43.777100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.777092    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnb87\" (UniqueName: \"kubernetes.io/projected/2f468c53-9680-43fd-ab80-e6bd9c7761bb-kube-api-access-qnb87\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:34:43.777100 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.777110    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:34:43.777326 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.777120    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f468c53-9680-43fd-ab80-e6bd9c7761bb-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:34:43.808186 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.808095    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f468c53-9680-43fd-ab80-e6bd9c7761bb" path="/var/lib/kubelet/pods/2f468c53-9680-43fd-ab80-e6bd9c7761bb/volumes"
Sep 15 12:34:43.943742 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:43.943709    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:45.620291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:45.620254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:45.724457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:45.724426    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:48.256209 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:48.256170    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:48.356155 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:48.356117    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:49.999525 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:49.999493    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:50.101306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:50.101270    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:50.286803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:50.286705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:54.391663 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:54.391566    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:55.088916 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:55.088875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:55.889687 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:55.889653    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:56.544096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:56.544059    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:57.208355 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:57.208320    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:57.900179 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:57.900138    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:59.001129 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:34:59.001091    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:09.711575 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.711542    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"]
Sep 15 12:35:09.712001 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.711766    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz" podUID="3af07cef-d1fc-41cd-8fac-5855008c7fff" containerName="maas-api" containerID="cri-o://e39a44ad7ca2fbcfaaabfa06c65e3a9656f7811ef2be44a232a6175dfd4d9b9b" gracePeriod=30
Sep 15 12:35:09.747615 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.747579    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:09.820162 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.820126    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:09.848839 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.848805    2583 generic.go:356] "Generic (PLEG): container finished" podID="3af07cef-d1fc-41cd-8fac-5855008c7fff" containerID="e39a44ad7ca2fbcfaaabfa06c65e3a9656f7811ef2be44a232a6175dfd4d9b9b" exitCode=0
Sep 15 12:35:09.849071 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.848883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz" event={"ID":"3af07cef-d1fc-41cd-8fac-5855008c7fff","Type":"ContainerDied","Data":"e39a44ad7ca2fbcfaaabfa06c65e3a9656f7811ef2be44a232a6175dfd4d9b9b"}
Sep 15 12:35:09.898005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.897965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:09.965264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.965210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:09.979309 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:09.979283    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:35:10.038099 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.038063    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3af07cef-d1fc-41cd-8fac-5855008c7fff-openshift-service-ca\") pod \"3af07cef-d1fc-41cd-8fac-5855008c7fff\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") "
Sep 15 12:35:10.038099 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.038098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-maas-api-tls\") pod \"3af07cef-d1fc-41cd-8fac-5855008c7fff\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") "
Sep 15 12:35:10.038371 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.038131    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2llv\" (UniqueName: \"kubernetes.io/projected/3af07cef-d1fc-41cd-8fac-5855008c7fff-kube-api-access-r2llv\") pod \"3af07cef-d1fc-41cd-8fac-5855008c7fff\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") "
Sep 15 12:35:10.038371 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.038235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-metrics-certs\") pod \"3af07cef-d1fc-41cd-8fac-5855008c7fff\" (UID: \"3af07cef-d1fc-41cd-8fac-5855008c7fff\") "
Sep 15 12:35:10.038520 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.038481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af07cef-d1fc-41cd-8fac-5855008c7fff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3af07cef-d1fc-41cd-8fac-5855008c7fff" (UID: "3af07cef-d1fc-41cd-8fac-5855008c7fff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:35:10.038581 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.038568    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3af07cef-d1fc-41cd-8fac-5855008c7fff-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:35:10.040611 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.040580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af07cef-d1fc-41cd-8fac-5855008c7fff-kube-api-access-r2llv" (OuterVolumeSpecName: "kube-api-access-r2llv") pod "3af07cef-d1fc-41cd-8fac-5855008c7fff" (UID: "3af07cef-d1fc-41cd-8fac-5855008c7fff"). InnerVolumeSpecName "kube-api-access-r2llv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:35:10.042342 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.042314    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3af07cef-d1fc-41cd-8fac-5855008c7fff" (UID: "3af07cef-d1fc-41cd-8fac-5855008c7fff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:35:10.042465 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.042425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3af07cef-d1fc-41cd-8fac-5855008c7fff" (UID: "3af07cef-d1fc-41cd-8fac-5855008c7fff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:35:10.139751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.139705    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:35:10.139751 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.139748    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3af07cef-d1fc-41cd-8fac-5855008c7fff-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:35:10.140003 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.139763    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2llv\" (UniqueName: \"kubernetes.io/projected/3af07cef-d1fc-41cd-8fac-5855008c7fff-kube-api-access-r2llv\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:35:10.379552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.379515    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:10.478791 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.478750    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:10.855906 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.855877    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"
Sep 15 12:35:10.856366 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.855873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz" event={"ID":"3af07cef-d1fc-41cd-8fac-5855008c7fff","Type":"ContainerDied","Data":"9b140f48e9bc4142f14000f44a7f2467f07cd25ddec084fa58299e7fe41c2a4d"}
Sep 15 12:35:10.856366 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.856016    2583 scope.go:117] "RemoveContainer" containerID="e39a44ad7ca2fbcfaaabfa06c65e3a9656f7811ef2be44a232a6175dfd4d9b9b"
Sep 15 12:35:10.878256 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.878229    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"]
Sep 15 12:35:10.882437 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:10.882411    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ed21e2-7ccb5f9978-6g2bz"]
Sep 15 12:35:11.093339 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:11.093306    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:11.763477 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:11.763442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:11.807829 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:11.807796    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af07cef-d1fc-41cd-8fac-5855008c7fff" path="/var/lib/kubelet/pods/3af07cef-d1fc-41cd-8fac-5855008c7fff/volumes"
Sep 15 12:35:12.597422 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:12.597392    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:17.764211 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:17.764170    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:52.342570 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:52.342530    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:52.560404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:52.560359    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:58.338876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:58.338837    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:58.442188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:58.442145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:58.603467 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:58.603368    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"]
Sep 15 12:35:58.603702 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:58.603675    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy" containerID="cri-o://5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8" gracePeriod=30
Sep 15 12:35:58.630619 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:58.630577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:59.200596 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:35:59.200551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:00.095412 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:00.095366    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:00.982783 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:00.982737    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:01.584360 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:01.584317    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:02.313535 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:02.313492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:03.864598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.864534    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:36:03.938318 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938283    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-token\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-credential-socket\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-socket\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4djgs\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-kube-api-access-4djgs\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-certs\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-envoy\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938527    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27dea5d-1da6-4b11-8f1b-278e95154db6-istiod-ca-cert\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-data\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938602    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:03.938754 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-podinfo\") pod \"e27dea5d-1da6-4b11-8f1b-278e95154db6\" (UID: \"e27dea5d-1da6-4b11-8f1b-278e95154db6\") "
Sep 15 12:36:03.938974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:03.938974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:03.938974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.938944    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27dea5d-1da6-4b11-8f1b-278e95154db6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:03.939142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.939064    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-data" (OuterVolumeSpecName: "istio-data") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:03.939195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.939153    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27dea5d-1da6-4b11-8f1b-278e95154db6-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:03.939195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.939173    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:03.939195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.939184    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:03.939195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.939193    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:03.939321 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.939202    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:03.940669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.940642    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-kube-api-access-4djgs" (OuterVolumeSpecName: "kube-api-access-4djgs") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "kube-api-access-4djgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:03.940820 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.940800    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:36:03.941040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.941008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:03.941152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:03.941055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-token" (OuterVolumeSpecName: "istio-token") pod "e27dea5d-1da6-4b11-8f1b-278e95154db6" (UID: "e27dea5d-1da6-4b11-8f1b-278e95154db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:04.040040 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.039998    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:04.040217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.040061    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:04.040217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.040078    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:04.040217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.040087    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4djgs\" (UniqueName: \"kubernetes.io/projected/e27dea5d-1da6-4b11-8f1b-278e95154db6-kube-api-access-4djgs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:04.146459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.146372    2583 generic.go:356] "Generic (PLEG): container finished" podID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerID="5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8" exitCode=0
Sep 15 12:36:04.146459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.146428    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"
Sep 15 12:36:04.146641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.146456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" event={"ID":"e27dea5d-1da6-4b11-8f1b-278e95154db6","Type":"ContainerDied","Data":"5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8"}
Sep 15 12:36:04.146641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.146516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs" event={"ID":"e27dea5d-1da6-4b11-8f1b-278e95154db6","Type":"ContainerDied","Data":"40d1f1a86060e253a5e701d7e4ac986b3a9ae524da75b7f7485f404ec040903b"}
Sep 15 12:36:04.146641 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.146537    2583 scope.go:117] "RemoveContainer" containerID="5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8"
Sep 15 12:36:04.156358 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.156339    2583 scope.go:117] "RemoveContainer" containerID="55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"
Sep 15 12:36:04.164798 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.164779    2583 scope.go:117] "RemoveContainer" containerID="5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8"
Sep 15 12:36:04.165102 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:04.165076    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8\": container with ID starting with 5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8 not found: ID does not exist" containerID="5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8"
Sep 15 12:36:04.165188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.165116    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8"} err="failed to get container status \"5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8\": rpc error: code = NotFound desc = could not find container \"5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8\": container with ID starting with 5ef560c773344f0cdbbb84167121b77f8db5cd95e580467da27bd61d1e58c7d8 not found: ID does not exist"
Sep 15 12:36:04.165188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.165141    2583 scope.go:117] "RemoveContainer" containerID="55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"
Sep 15 12:36:04.165403 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:04.165386    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade\": container with ID starting with 55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade not found: ID does not exist" containerID="55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"
Sep 15 12:36:04.165448 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.165409    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade"} err="failed to get container status \"55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade\": rpc error: code = NotFound desc = could not find container \"55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade\": container with ID starting with 55414fc29fafeb0d687d31fa97b8023b2bdfd49251da22b16961be8019b48ade not found: ID does not exist"
Sep 15 12:36:04.169870 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.169845    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"]
Sep 15 12:36:04.173851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:04.173830    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ed21e2-openshift-default-7f68cb57bd-r6pxs"]
Sep 15 12:36:05.807528 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:05.807495    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" path="/var/lib/kubelet/pods/e27dea5d-1da6-4b11-8f1b-278e95154db6/volumes"
Sep 15 12:36:06.314974 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:06.314940    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:09.505771 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.505737    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"]
Sep 15 12:36:09.506294 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.506050    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr" podUID="d01a2111-500b-4e67-ba7e-75300b02e040" containerName="maas-api" containerID="cri-o://201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de" gracePeriod=30
Sep 15 12:36:09.544131 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.544093    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:09.644581 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.644538    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:09.732598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.732558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:09.758802 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.758743    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:36:09.795876 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.795845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6z42\" (UniqueName: \"kubernetes.io/projected/d01a2111-500b-4e67-ba7e-75300b02e040-kube-api-access-l6z42\") pod \"d01a2111-500b-4e67-ba7e-75300b02e040\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") "
Sep 15 12:36:09.796069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.795887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-maas-api-tls\") pod \"d01a2111-500b-4e67-ba7e-75300b02e040\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") "
Sep 15 12:36:09.796069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.796018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01a2111-500b-4e67-ba7e-75300b02e040-openshift-service-ca\") pod \"d01a2111-500b-4e67-ba7e-75300b02e040\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") "
Sep 15 12:36:09.796203 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.796109    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-metrics-certs\") pod \"d01a2111-500b-4e67-ba7e-75300b02e040\" (UID: \"d01a2111-500b-4e67-ba7e-75300b02e040\") "
Sep 15 12:36:09.796368 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.796340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01a2111-500b-4e67-ba7e-75300b02e040-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d01a2111-500b-4e67-ba7e-75300b02e040" (UID: "d01a2111-500b-4e67-ba7e-75300b02e040"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:09.796473 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.796456    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01a2111-500b-4e67-ba7e-75300b02e040-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:09.798249 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.798211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d01a2111-500b-4e67-ba7e-75300b02e040" (UID: "d01a2111-500b-4e67-ba7e-75300b02e040"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:09.798831 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.798798    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:09.798947 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.798801    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d01a2111-500b-4e67-ba7e-75300b02e040" (UID: "d01a2111-500b-4e67-ba7e-75300b02e040"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:09.799096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.799075    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01a2111-500b-4e67-ba7e-75300b02e040-kube-api-access-l6z42" (OuterVolumeSpecName: "kube-api-access-l6z42") pod "d01a2111-500b-4e67-ba7e-75300b02e040" (UID: "d01a2111-500b-4e67-ba7e-75300b02e040"). InnerVolumeSpecName "kube-api-access-l6z42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:09.897382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.897346    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:09.897382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.897382    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6z42\" (UniqueName: \"kubernetes.io/projected/d01a2111-500b-4e67-ba7e-75300b02e040-kube-api-access-l6z42\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:09.897382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:09.897398    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01a2111-500b-4e67-ba7e-75300b02e040-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:10.133598 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.133549    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:10.181565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.181526    2583 generic.go:356] "Generic (PLEG): container finished" podID="d01a2111-500b-4e67-ba7e-75300b02e040" containerID="201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de" exitCode=0
Sep 15 12:36:10.181764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.181577    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"
Sep 15 12:36:10.181764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.181599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr" event={"ID":"d01a2111-500b-4e67-ba7e-75300b02e040","Type":"ContainerDied","Data":"201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de"}
Sep 15 12:36:10.181764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.181653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr" event={"ID":"d01a2111-500b-4e67-ba7e-75300b02e040","Type":"ContainerDied","Data":"06f76b9cd30cabeeb13bda3cb94803cd5e38bf026c67f86ed0ccde0c5c5038a5"}
Sep 15 12:36:10.181764 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.181683    2583 scope.go:117] "RemoveContainer" containerID="201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de"
Sep 15 12:36:10.191352 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.191333    2583 scope.go:117] "RemoveContainer" containerID="201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de"
Sep 15 12:36:10.191647 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:10.191630    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de\": container with ID starting with 201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de not found: ID does not exist" containerID="201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de"
Sep 15 12:36:10.191702 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.191655    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de"} err="failed to get container status \"201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de\": rpc error: code = NotFound desc = could not find container \"201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de\": container with ID starting with 201888be95e3f2e435b4d1f4ed76e004b2948937d8590e446aaedd0c1557c5de not found: ID does not exist"
Sep 15 12:36:10.206566 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.206535    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"]
Sep 15 12:36:10.209368 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.209338    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7b204e-666cdf46c8-jw5lr"]
Sep 15 12:36:10.224791 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:10.224745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:11.807739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:11.807705    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d01a2111-500b-4e67-ba7e-75300b02e040" path="/var/lib/kubelet/pods/d01a2111-500b-4e67-ba7e-75300b02e040/volumes"
Sep 15 12:36:13.515879 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:13.515850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:13.617934 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:13.617893    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:15.932468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:15.932412    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:15.995015 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:15.994979    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:16.224708 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:16.224616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:16.717221 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:16.717179    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:17.589018 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:17.588981    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:18.416733 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:18.416661    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:19.144522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:19.144474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:27.092311 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.092169    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"]
Sep 15 12:36:27.093010 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.092963    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" podUID="d448aa45-786d-474e-a9eb-19db08677274" containerName="maas-api" containerID="cri-o://da38958427e0c79d033ac10f50384086533ac615f7a341fe62d077bddbd8d477" gracePeriod=30
Sep 15 12:36:27.116087 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.116045    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:27.213595 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.213558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:27.275472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.275442    2583 generic.go:356] "Generic (PLEG): container finished" podID="d448aa45-786d-474e-a9eb-19db08677274" containerID="da38958427e0c79d033ac10f50384086533ac615f7a341fe62d077bddbd8d477" exitCode=0
Sep 15 12:36:27.275622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.275479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" event={"ID":"d448aa45-786d-474e-a9eb-19db08677274","Type":"ContainerDied","Data":"da38958427e0c79d033ac10f50384086533ac615f7a341fe62d077bddbd8d477"}
Sep 15 12:36:27.315960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.315914    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:27.348765 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.348702    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:36:27.382933 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.382897    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:27.475981 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.475942    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6wmw\" (UniqueName: \"kubernetes.io/projected/d448aa45-786d-474e-a9eb-19db08677274-kube-api-access-r6wmw\") pod \"d448aa45-786d-474e-a9eb-19db08677274\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") "
Sep 15 12:36:27.476183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.476076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-metrics-certs\") pod \"d448aa45-786d-474e-a9eb-19db08677274\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") "
Sep 15 12:36:27.476183 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.476148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") pod \"d448aa45-786d-474e-a9eb-19db08677274\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") "
Sep 15 12:36:27.476306 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.476190    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448aa45-786d-474e-a9eb-19db08677274-openshift-service-ca\") pod \"d448aa45-786d-474e-a9eb-19db08677274\" (UID: \"d448aa45-786d-474e-a9eb-19db08677274\") "
Sep 15 12:36:27.477286 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.477249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d448aa45-786d-474e-a9eb-19db08677274-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d448aa45-786d-474e-a9eb-19db08677274" (UID: "d448aa45-786d-474e-a9eb-19db08677274"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:27.483752 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.483717    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d448aa45-786d-474e-a9eb-19db08677274" (UID: "d448aa45-786d-474e-a9eb-19db08677274"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:27.485257 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.485232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d448aa45-786d-474e-a9eb-19db08677274" (UID: "d448aa45-786d-474e-a9eb-19db08677274"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:27.486809 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.486774    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d448aa45-786d-474e-a9eb-19db08677274-kube-api-access-r6wmw" (OuterVolumeSpecName: "kube-api-access-r6wmw") pod "d448aa45-786d-474e-a9eb-19db08677274" (UID: "d448aa45-786d-474e-a9eb-19db08677274"). InnerVolumeSpecName "kube-api-access-r6wmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:27.578555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.578512    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:27.578555 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.578549    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d448aa45-786d-474e-a9eb-19db08677274-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:27.578784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.578581    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d448aa45-786d-474e-a9eb-19db08677274-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:27.578784 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.578591    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6wmw\" (UniqueName: \"kubernetes.io/projected/d448aa45-786d-474e-a9eb-19db08677274-kube-api-access-r6wmw\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:27.710068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.709950    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:27.810423 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:27.810381    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:28.282557 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:28.282515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f" event={"ID":"d448aa45-786d-474e-a9eb-19db08677274","Type":"ContainerDied","Data":"6864bab9421a6dc2d424ecfadd098c406a21cbabc12b31f388d94e0398c32dd0"}
Sep 15 12:36:28.282557 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:28.282557    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"
Sep 15 12:36:28.283094 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:28.282585    2583 scope.go:117] "RemoveContainer" containerID="da38958427e0c79d033ac10f50384086533ac615f7a341fe62d077bddbd8d477"
Sep 15 12:36:28.299291 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:28.299262    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"]
Sep 15 12:36:28.301182 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:28.301161    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1dc430-6994fcf48f-t7z9f"]
Sep 15 12:36:28.502280 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:28.502238    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:29.808088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:29.808052    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d448aa45-786d-474e-a9eb-19db08677274" path="/var/lib/kubelet/pods/d448aa45-786d-474e-a9eb-19db08677274/volumes"
Sep 15 12:36:33.629792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:33.629759    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:33.732459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:33.732427    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:33.964296 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:33.964211    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"]
Sep 15 12:36:33.964471 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:33.964448    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy" containerID="cri-o://d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5" gracePeriod=30
Sep 15 12:36:35.605893 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:35.605853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:35.706357 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:35.706319    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:36.094323 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:36.094286    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:39.223475 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.223450    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:36:39.284757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-data\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.284757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284719    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-istio-token\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.284757 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8dd5701-61f4-4956-911f-9af19db86494-istio-podinfo\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-socket\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284835    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-certs\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284901    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kx26\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-kube-api-access-9kx26\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8dd5701-61f4-4956-911f-9af19db86494-istiod-ca-cert\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284982    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-envoy\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.284985    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:39.285069 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285066    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-credential-socket\") pod \"a8dd5701-61f4-4956-911f-9af19db86494\" (UID: \"a8dd5701-61f4-4956-911f-9af19db86494\") "
Sep 15 12:36:39.285438 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:39.285492 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285428    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:39.285492 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285445    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dd5701-61f4-4956-911f-9af19db86494-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:39.285591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285489    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.285591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285510    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.285591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.285533    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:39.287016 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.286996    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8dd5701-61f4-4956-911f-9af19db86494-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:36:39.287135 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.287119    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:39.287536 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.287514    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:39.287600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.287546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-kube-api-access-9kx26" (OuterVolumeSpecName: "kube-api-access-9kx26") pod "a8dd5701-61f4-4956-911f-9af19db86494" (UID: "a8dd5701-61f4-4956-911f-9af19db86494"). InnerVolumeSpecName "kube-api-access-9kx26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:39.344123 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.344089    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8dd5701-61f4-4956-911f-9af19db86494" containerID="d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5" exitCode=0
Sep 15 12:36:39.344285 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.344134    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"
Sep 15 12:36:39.344285 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.344148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" event={"ID":"a8dd5701-61f4-4956-911f-9af19db86494","Type":"ContainerDied","Data":"d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5"}
Sep 15 12:36:39.344285 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.344224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w" event={"ID":"a8dd5701-61f4-4956-911f-9af19db86494","Type":"ContainerDied","Data":"2b66d13a64fef50f33f7bfe8c7e3c3cd54c2fa6d3ba8a8b664a53c1ae1ede68f"}
Sep 15 12:36:39.344285 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.344247    2583 scope.go:117] "RemoveContainer" containerID="d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5"
Sep 15 12:36:39.354278 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.354262    2583 scope.go:117] "RemoveContainer" containerID="19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"
Sep 15 12:36:39.362732 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.362712    2583 scope.go:117] "RemoveContainer" containerID="d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5"
Sep 15 12:36:39.363017 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:39.362995    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5\": container with ID starting with d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5 not found: ID does not exist" containerID="d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5"
Sep 15 12:36:39.363108 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.363035    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5"} err="failed to get container status \"d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5\": rpc error: code = NotFound desc = could not find container \"d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5\": container with ID starting with d9800b68de581b3e1418bb1d183cc9851eddb8686e0f98e5b517585eff42e7d5 not found: ID does not exist"
Sep 15 12:36:39.363108 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.363055    2583 scope.go:117] "RemoveContainer" containerID="19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"
Sep 15 12:36:39.363293 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:39.363275    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a\": container with ID starting with 19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a not found: ID does not exist" containerID="19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"
Sep 15 12:36:39.363337 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.363297    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a"} err="failed to get container status \"19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a\": rpc error: code = NotFound desc = could not find container \"19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a\": container with ID starting with 19064b51cd2e422385985dfb1db69e52ddc2acf0d9f73d11b216718ac481c00a not found: ID does not exist"
Sep 15 12:36:39.367033 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.366999    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"]
Sep 15 12:36:39.371280 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.371257    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1dc430-openshift-default-56c797c478-jzw2w"]
Sep 15 12:36:39.385950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385924    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kx26\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-kube-api-access-9kx26\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.385950 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385951    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8dd5701-61f4-4956-911f-9af19db86494-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.386142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385960    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.386142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385969    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.386142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385978    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8dd5701-61f4-4956-911f-9af19db86494-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.386142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385986    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8dd5701-61f4-4956-911f-9af19db86494-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.386142 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.385993    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8dd5701-61f4-4956-911f-9af19db86494-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:39.806715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:39.806683    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8dd5701-61f4-4956-911f-9af19db86494" path="/var/lib/kubelet/pods/a8dd5701-61f4-4956-911f-9af19db86494/volumes"
Sep 15 12:36:41.709787 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:41.709747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:41.810270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:41.810232    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:44.704295 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:44.704262    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"]
Sep 15 12:36:44.704741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:44.704475    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" podUID="76ab737a-9996-4d0c-bf36-bb397f96b45e" containerName="maas-api" containerID="cri-o://a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643" gracePeriod=30
Sep 15 12:36:44.737087 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:44.737054    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:44.836301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:44.836265    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:44.936803 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:44.936772    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:44.965468 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:44.965398    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:36:45.036303 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.036272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ab737a-9996-4d0c-bf36-bb397f96b45e-openshift-service-ca\") pod \"76ab737a-9996-4d0c-bf36-bb397f96b45e\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") "
Sep 15 12:36:45.036502 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.036342    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-metrics-certs\") pod \"76ab737a-9996-4d0c-bf36-bb397f96b45e\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") "
Sep 15 12:36:45.036502 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.036433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94qk9\" (UniqueName: \"kubernetes.io/projected/76ab737a-9996-4d0c-bf36-bb397f96b45e-kube-api-access-94qk9\") pod \"76ab737a-9996-4d0c-bf36-bb397f96b45e\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") "
Sep 15 12:36:45.036502 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.036468    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") pod \"76ab737a-9996-4d0c-bf36-bb397f96b45e\" (UID: \"76ab737a-9996-4d0c-bf36-bb397f96b45e\") "
Sep 15 12:36:45.036993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.036868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ab737a-9996-4d0c-bf36-bb397f96b45e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76ab737a-9996-4d0c-bf36-bb397f96b45e" (UID: "76ab737a-9996-4d0c-bf36-bb397f96b45e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:45.036993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.036947    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:45.038797 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.038770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76ab737a-9996-4d0c-bf36-bb397f96b45e" (UID: "76ab737a-9996-4d0c-bf36-bb397f96b45e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:45.038886 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.038824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76ab737a-9996-4d0c-bf36-bb397f96b45e" (UID: "76ab737a-9996-4d0c-bf36-bb397f96b45e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:45.038935 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.038914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ab737a-9996-4d0c-bf36-bb397f96b45e-kube-api-access-94qk9" (OuterVolumeSpecName: "kube-api-access-94qk9") pod "76ab737a-9996-4d0c-bf36-bb397f96b45e" (UID: "76ab737a-9996-4d0c-bf36-bb397f96b45e"). InnerVolumeSpecName "kube-api-access-94qk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:45.137382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.137345    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:45.137382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.137378    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qk9\" (UniqueName: \"kubernetes.io/projected/76ab737a-9996-4d0c-bf36-bb397f96b45e-kube-api-access-94qk9\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:45.137382 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.137389    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76ab737a-9996-4d0c-bf36-bb397f96b45e-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:45.137616 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.137398    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ab737a-9996-4d0c-bf36-bb397f96b45e-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:45.336246 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.336213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:45.385435 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.385396    2583 generic.go:356] "Generic (PLEG): container finished" podID="76ab737a-9996-4d0c-bf36-bb397f96b45e" containerID="a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643" exitCode=0
Sep 15 12:36:45.385589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.385441    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"
Sep 15 12:36:45.385589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.385480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" event={"ID":"76ab737a-9996-4d0c-bf36-bb397f96b45e","Type":"ContainerDied","Data":"a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643"}
Sep 15 12:36:45.385589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.385529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb" event={"ID":"76ab737a-9996-4d0c-bf36-bb397f96b45e","Type":"ContainerDied","Data":"7c19612c94b75b7b4ac60710847fc2dfcea1c3fa10086084d97341e02015492f"}
Sep 15 12:36:45.385589 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.385550    2583 scope.go:117] "RemoveContainer" containerID="a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643"
Sep 15 12:36:45.395402 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.395374    2583 scope.go:117] "RemoveContainer" containerID="a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643"
Sep 15 12:36:45.395682 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:45.395661    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643\": container with ID starting with a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643 not found: ID does not exist" containerID="a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643"
Sep 15 12:36:45.395750 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.395691    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643"} err="failed to get container status \"a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643\": rpc error: code = NotFound desc = could not find container \"a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643\": container with ID starting with a68d9ee6a416d93387ad08f04cc626e51ad78e1ea07689581ff96613e585c643 not found: ID does not exist"
Sep 15 12:36:45.406833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.406811    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"]
Sep 15 12:36:45.410351 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.410323    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:45.410900 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.410880    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-969130-68cdb7bf87-k54gb"]
Sep 15 12:36:45.807864 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:45.807769    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76ab737a-9996-4d0c-bf36-bb397f96b45e" path="/var/lib/kubelet/pods/76ab737a-9996-4d0c-bf36-bb397f96b45e/volumes"
Sep 15 12:36:51.100457 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:51.100417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:51.174469 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:51.174429    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:51.367573 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:51.367474    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"]
Sep 15 12:36:51.367746 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:51.367720    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" containerID="cri-o://6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399" gracePeriod=30
Sep 15 12:36:51.395450 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:51.395410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:54.339201 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:54.339158    2583 patch_prober.go:28] interesting pod/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 12:36:54.339582 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:54.339243    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:36:55.039887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.039848    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-4cddr"]
Sep 15 12:36:55.040147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.040075    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr" podUID="fbf42d58-7d87-404c-b210-c4fd9abced30" containerName="manager" containerID="cri-o://2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15" gracePeriod=30
Sep 15 12:36:55.284597 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.284573    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:36:55.432701 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.432664    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz92s\" (UniqueName: \"kubernetes.io/projected/fbf42d58-7d87-404c-b210-c4fd9abced30-kube-api-access-jz92s\") pod \"fbf42d58-7d87-404c-b210-c4fd9abced30\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") "
Sep 15 12:36:55.433154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.432740    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert\") pod \"fbf42d58-7d87-404c-b210-c4fd9abced30\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") "
Sep 15 12:36:55.433154 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.432777    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs\") pod \"fbf42d58-7d87-404c-b210-c4fd9abced30\" (UID: \"fbf42d58-7d87-404c-b210-c4fd9abced30\") "
Sep 15 12:36:55.435013 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.434984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf42d58-7d87-404c-b210-c4fd9abced30-kube-api-access-jz92s" (OuterVolumeSpecName: "kube-api-access-jz92s") pod "fbf42d58-7d87-404c-b210-c4fd9abced30" (UID: "fbf42d58-7d87-404c-b210-c4fd9abced30"). InnerVolumeSpecName "kube-api-access-jz92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:55.435013 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.434983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbf42d58-7d87-404c-b210-c4fd9abced30" (UID: "fbf42d58-7d87-404c-b210-c4fd9abced30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:55.435215 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.435199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fbf42d58-7d87-404c-b210-c4fd9abced30" (UID: "fbf42d58-7d87-404c-b210-c4fd9abced30"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:36:55.439012 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.438987    2583 generic.go:356] "Generic (PLEG): container finished" podID="fbf42d58-7d87-404c-b210-c4fd9abced30" containerID="2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15" exitCode=0
Sep 15 12:36:55.439139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.439053    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr"
Sep 15 12:36:55.439139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.439052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr" event={"ID":"fbf42d58-7d87-404c-b210-c4fd9abced30","Type":"ContainerDied","Data":"2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15"}
Sep 15 12:36:55.439139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.439116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-4cddr" event={"ID":"fbf42d58-7d87-404c-b210-c4fd9abced30","Type":"ContainerDied","Data":"c09f92f43bf349c3462e7b8bbff911fed5e1a4981798a2600f817d7b143c0432"}
Sep 15 12:36:55.439245 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.439149    2583 scope.go:117] "RemoveContainer" containerID="2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15"
Sep 15 12:36:55.454144 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.454115    2583 scope.go:117] "RemoveContainer" containerID="2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15"
Sep 15 12:36:55.454385 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:55.454364    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15\": container with ID starting with 2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15 not found: ID does not exist" containerID="2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15"
Sep 15 12:36:55.454434 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.454393    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15"} err="failed to get container status \"2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15\": rpc error: code = NotFound desc = could not find container \"2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15\": container with ID starting with 2164ecc2d4e95dae883d211433774c38fb33dc50e4154161f0c67d9a3c864f15 not found: ID does not exist"
Sep 15 12:36:55.468264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.468239    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-4cddr"]
Sep 15 12:36:55.477599 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.477574    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-4cddr"]
Sep 15 12:36:55.533414 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.533380    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz92s\" (UniqueName: \"kubernetes.io/projected/fbf42d58-7d87-404c-b210-c4fd9abced30-kube-api-access-jz92s\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:55.533414 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.533413    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-webhook-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:55.533628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.533428    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf42d58-7d87-404c-b210-c4fd9abced30-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:55.807674 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:55.807588    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf42d58-7d87-404c-b210-c4fd9abced30" path="/var/lib/kubelet/pods/fbf42d58-7d87-404c-b210-c4fd9abced30/volumes"
Sep 15 12:36:56.627109 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.627081    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:36:56.744799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744705    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqt54\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-kube-api-access-lqt54\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.744799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-socket\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.744799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-podinfo\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.744799 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-token\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.745206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-data\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.745206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744945    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f141f617-e485-4035-b5e5-722c8e1d48b1-istiod-ca-cert\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.745206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.744972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-certs\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.745206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745009    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-envoy\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.745206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745132    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-credential-socket\") pod \"f141f617-e485-4035-b5e5-722c8e1d48b1\" (UID: \"f141f617-e485-4035-b5e5-722c8e1d48b1\") "
Sep 15 12:36:56.745206 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:56.745519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745231    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:56.745519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:56.745519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f141f617-e485-4035-b5e5-722c8e1d48b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:36:56.745519 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:56.745745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745581    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.745745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745597    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-data\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.745745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745605    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f141f617-e485-4035-b5e5-722c8e1d48b1-istiod-ca-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.745745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745613    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-workload-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.745745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.745620    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-credential-socket\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.747055 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.747007    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:56.747215 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.747201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:36:56.747384 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.747369    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-kube-api-access-lqt54" (OuterVolumeSpecName: "kube-api-access-lqt54") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "kube-api-access-lqt54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:36:56.747451 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.747429    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f141f617-e485-4035-b5e5-722c8e1d48b1" (UID: "f141f617-e485-4035-b5e5-722c8e1d48b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 12:36:56.846300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.846261    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-envoy\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.846300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.846293    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqt54\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-kube-api-access-lqt54\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.846300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.846305    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-podinfo\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:56.846300 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:56.846314    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f141f617-e485-4035-b5e5-722c8e1d48b1-istio-token\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:36:57.452441 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.452406    2583 generic.go:356] "Generic (PLEG): container finished" podID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerID="6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399" exitCode=0
Sep 15 12:36:57.452620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.452455    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"
Sep 15 12:36:57.452620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.452493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" event={"ID":"f141f617-e485-4035-b5e5-722c8e1d48b1","Type":"ContainerDied","Data":"6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399"}
Sep 15 12:36:57.452620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.452558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz" event={"ID":"f141f617-e485-4035-b5e5-722c8e1d48b1","Type":"ContainerDied","Data":"4194c5d0ef85c0160eaa90d59b7e8055f6416b000aa63c308c1831bc5cd7ae28"}
Sep 15 12:36:57.452620 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.452587    2583 scope.go:117] "RemoveContainer" containerID="6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399"
Sep 15 12:36:57.462477 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.462461    2583 scope.go:117] "RemoveContainer" containerID="1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"
Sep 15 12:36:57.470966 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.470948    2583 scope.go:117] "RemoveContainer" containerID="6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399"
Sep 15 12:36:57.471261 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:57.471241    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399\": container with ID starting with 6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399 not found: ID does not exist" containerID="6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399"
Sep 15 12:36:57.471338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.471274    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399"} err="failed to get container status \"6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399\": rpc error: code = NotFound desc = could not find container \"6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399\": container with ID starting with 6b1dc08fb264a7b798a2775efa8fd5178c65368634b901cf245f164103651399 not found: ID does not exist"
Sep 15 12:36:57.471338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.471297    2583 scope.go:117] "RemoveContainer" containerID="1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"
Sep 15 12:36:57.471608 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:36:57.471577    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d\": container with ID starting with 1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d not found: ID does not exist" containerID="1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"
Sep 15 12:36:57.471712 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.471609    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d"} err="failed to get container status \"1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d\": rpc error: code = NotFound desc = could not find container \"1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d\": container with ID starting with 1c8fac70659c13c91622b694952a3bddcb120027249a64a6667fba63c04d949d not found: ID does not exist"
Sep 15 12:36:57.478046 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.475558    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"]
Sep 15 12:36:57.482281 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.482258    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-969130-openshift-default-694d7cfc78-hrtzz"]
Sep 15 12:36:57.806299 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:57.806201    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" path="/var/lib/kubelet/pods/f141f617-e485-4035-b5e5-722c8e1d48b1/volumes"
Sep 15 12:36:59.924515 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:59.924488    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:36:59.930193 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:36:59.930169    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:37:07.459579 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:07.459539    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:37:56.302880 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.302801    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-c5xdk"]
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303240    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af07cef-d1fc-41cd-8fac-5855008c7fff" containerName="maas-api"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303256    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af07cef-d1fc-41cd-8fac-5855008c7fff" containerName="maas-api"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303266    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76ab737a-9996-4d0c-bf36-bb397f96b45e" containerName="maas-api"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303271    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="76ab737a-9996-4d0c-bf36-bb397f96b45e" containerName="maas-api"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303280    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf42d58-7d87-404c-b210-c4fd9abced30" containerName="manager"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303286    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf42d58-7d87-404c-b210-c4fd9abced30" containerName="manager"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303293    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d448aa45-786d-474e-a9eb-19db08677274" containerName="maas-api"
Sep 15 12:37:56.303301 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303298    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d448aa45-786d-474e-a9eb-19db08677274" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303315    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303320    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303327    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303332    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303338    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f468c53-9680-43fd-ab80-e6bd9c7761bb" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f468c53-9680-43fd-ab80-e6bd9c7761bb" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303355    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01a2111-500b-4e67-ba7e-75300b02e040" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303360    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01a2111-500b-4e67-ba7e-75300b02e040" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303365    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303370    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303376    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303381    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303387    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303393    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303449    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af07cef-d1fc-41cd-8fac-5855008c7fff" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303456    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303462    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303467    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d01a2111-500b-4e67-ba7e-75300b02e040" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303473    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf42d58-7d87-404c-b210-c4fd9abced30" containerName="manager"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303479    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303485    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f141f617-e485-4035-b5e5-722c8e1d48b1" containerName="istio-proxy"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303490    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="76ab737a-9996-4d0c-bf36-bb397f96b45e" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303496    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d448aa45-786d-474e-a9eb-19db08677274" containerName="maas-api"
Sep 15 12:37:56.303552 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.303502    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f468c53-9680-43fd-ab80-e6bd9c7761bb" containerName="maas-api"
Sep 15 12:37:56.306538 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.306521    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.308967 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.308938    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 12:37:56.309113 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.309066    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 12:37:56.309940 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.309919    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 12:37:56.309987 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.309963    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pb5qc\""
Sep 15 12:37:56.314697 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.314676    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-c5xdk"]
Sep 15 12:37:56.484505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.484464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-metrics-certs\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.484505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.484504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrstz\" (UniqueName: \"kubernetes.io/projected/adc84c58-0960-498c-b00e-35269a414770-kube-api-access-hrstz\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.484747 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.484664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-webhook-cert\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.586061 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.586002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-webhook-cert\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.586258 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.586068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-metrics-certs\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.586258 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.586090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrstz\" (UniqueName: \"kubernetes.io/projected/adc84c58-0960-498c-b00e-35269a414770-kube-api-access-hrstz\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.588553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.588526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-metrics-certs\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.588670 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.588590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-webhook-cert\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.594166 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.594146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrstz\" (UniqueName: \"kubernetes.io/projected/adc84c58-0960-498c-b00e-35269a414770-kube-api-access-hrstz\") pod \"maas-controller-bc7f7d7c4-c5xdk\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") " pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.618188 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.618157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:56.890914 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.890838    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-c5xdk"]
Sep 15 12:37:56.894157 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:37:56.894131    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc84c58_0960_498c_b00e_35269a414770.slice/crio-ee9bb6002b9a7c57461657486be15a380810a156750fa973b3abb4ea3086311c WatchSource:0}: Error finding container ee9bb6002b9a7c57461657486be15a380810a156750fa973b3abb4ea3086311c: Status 404 returned error can't find the container with id ee9bb6002b9a7c57461657486be15a380810a156750fa973b3abb4ea3086311c
Sep 15 12:37:56.895566 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:56.895546    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:37:57.763097 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:57.762979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk" event={"ID":"adc84c58-0960-498c-b00e-35269a414770","Type":"ContainerStarted","Data":"dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16"}
Sep 15 12:37:57.763097 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:57.763043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk" event={"ID":"adc84c58-0960-498c-b00e-35269a414770","Type":"ContainerStarted","Data":"ee9bb6002b9a7c57461657486be15a380810a156750fa973b3abb4ea3086311c"}
Sep 15 12:37:57.763565 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:57.763115    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:37:57.781910 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:37:57.781855    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk" podStartSLOduration=1.336274654 podStartE2EDuration="1.781838923s" totalImagesPullingTime="445.564269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:37:57.779595355 +0000 UTC m=+2158.476154954" watchObservedRunningTime="2026-09-15 12:37:57.781838923 +0000 UTC m=+2158.478398501"
Sep 15 12:38:08.770827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:08.770797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:38:15.900251 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:15.900216    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:15.997814 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:15.997766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-c5xdk"]
Sep 15 12:38:15.998130 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:15.998097    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk" podUID="adc84c58-0960-498c-b00e-35269a414770" containerName="manager" containerID="cri-o://dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16" gracePeriod=30
Sep 15 12:38:16.244998 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.244969    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:38:16.354534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.354486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-metrics-certs\") pod \"adc84c58-0960-498c-b00e-35269a414770\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") "
Sep 15 12:38:16.354715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.354577    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-webhook-cert\") pod \"adc84c58-0960-498c-b00e-35269a414770\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") "
Sep 15 12:38:16.354715 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.354638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrstz\" (UniqueName: \"kubernetes.io/projected/adc84c58-0960-498c-b00e-35269a414770-kube-api-access-hrstz\") pod \"adc84c58-0960-498c-b00e-35269a414770\" (UID: \"adc84c58-0960-498c-b00e-35269a414770\") "
Sep 15 12:38:16.356669 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.356637    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc84c58-0960-498c-b00e-35269a414770-kube-api-access-hrstz" (OuterVolumeSpecName: "kube-api-access-hrstz") pod "adc84c58-0960-498c-b00e-35269a414770" (UID: "adc84c58-0960-498c-b00e-35269a414770"). InnerVolumeSpecName "kube-api-access-hrstz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:38:16.356795 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.356771    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adc84c58-0960-498c-b00e-35269a414770" (UID: "adc84c58-0960-498c-b00e-35269a414770"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:38:16.356942 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.356916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "adc84c58-0960-498c-b00e-35269a414770" (UID: "adc84c58-0960-498c-b00e-35269a414770"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:38:16.456237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.456153    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:38:16.456237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.456180    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc84c58-0960-498c-b00e-35269a414770-webhook-cert\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:38:16.456237 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.456189    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrstz\" (UniqueName: \"kubernetes.io/projected/adc84c58-0960-498c-b00e-35269a414770-kube-api-access-hrstz\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:38:16.856873 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.856835    2583 generic.go:356] "Generic (PLEG): container finished" podID="adc84c58-0960-498c-b00e-35269a414770" containerID="dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16" exitCode=0
Sep 15 12:38:16.857068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.856884    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk"
Sep 15 12:38:16.857068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.856927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk" event={"ID":"adc84c58-0960-498c-b00e-35269a414770","Type":"ContainerDied","Data":"dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16"}
Sep 15 12:38:16.857068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.856982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-c5xdk" event={"ID":"adc84c58-0960-498c-b00e-35269a414770","Type":"ContainerDied","Data":"ee9bb6002b9a7c57461657486be15a380810a156750fa973b3abb4ea3086311c"}
Sep 15 12:38:16.857068 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.857004    2583 scope.go:117] "RemoveContainer" containerID="dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16"
Sep 15 12:38:16.866899 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.866880    2583 scope.go:117] "RemoveContainer" containerID="dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16"
Sep 15 12:38:16.867207 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:38:16.867190    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16\": container with ID starting with dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16 not found: ID does not exist" containerID="dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16"
Sep 15 12:38:16.867262 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.867216    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16"} err="failed to get container status \"dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16\": rpc error: code = NotFound desc = could not find container \"dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16\": container with ID starting with dbad365579b1d556378948e9190ee93b65374d32805473f7a6e1be0c798d5e16 not found: ID does not exist"
Sep 15 12:38:16.878264 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.878239    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-c5xdk"]
Sep 15 12:38:16.882360 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:16.882336    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-c5xdk"]
Sep 15 12:38:17.806695 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.806662    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc84c58-0960-498c-b00e-35269a414770" path="/var/lib/kubelet/pods/adc84c58-0960-498c-b00e-35269a414770/volumes"
Sep 15 12:38:17.994124 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994087    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-b4w5d"]
Sep 15 12:38:17.994532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994512    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy"
Sep 15 12:38:17.994628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994536    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy"
Sep 15 12:38:17.994628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994577    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc84c58-0960-498c-b00e-35269a414770" containerName="manager"
Sep 15 12:38:17.994628 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994587    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc84c58-0960-498c-b00e-35269a414770" containerName="manager"
Sep 15 12:38:17.994781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994679    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dd5701-61f4-4956-911f-9af19db86494" containerName="istio-proxy"
Sep 15 12:38:17.994781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994698    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc84c58-0960-498c-b00e-35269a414770" containerName="manager"
Sep 15 12:38:17.994781 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.994710    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27dea5d-1da6-4b11-8f1b-278e95154db6" containerName="istio-proxy"
Sep 15 12:38:17.999217 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:17.999195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.002359 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.002337    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 12:38:18.002359 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.002350    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pb5qc\""
Sep 15 12:38:18.002553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.002369    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 12:38:18.002553 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.002352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 12:38:18.012177 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.012151    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-b4w5d"]
Sep 15 12:38:18.067401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.067358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fe32e67-b5ae-40d5-9b7f-530f984faea2-webhook-cert\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.067401 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.067402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnbl\" (UniqueName: \"kubernetes.io/projected/2fe32e67-b5ae-40d5-9b7f-530f984faea2-kube-api-access-ffnbl\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.067623 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.067465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe32e67-b5ae-40d5-9b7f-530f984faea2-metrics-certs\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.167973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.167938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe32e67-b5ae-40d5-9b7f-530f984faea2-metrics-certs\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.168195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.168047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fe32e67-b5ae-40d5-9b7f-530f984faea2-webhook-cert\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.168195 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.168075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnbl\" (UniqueName: \"kubernetes.io/projected/2fe32e67-b5ae-40d5-9b7f-530f984faea2-kube-api-access-ffnbl\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.170514 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.170486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2fe32e67-b5ae-40d5-9b7f-530f984faea2-webhook-cert\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.170647 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.170512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe32e67-b5ae-40d5-9b7f-530f984faea2-metrics-certs\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.176650 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.176620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnbl\" (UniqueName: \"kubernetes.io/projected/2fe32e67-b5ae-40d5-9b7f-530f984faea2-kube-api-access-ffnbl\") pod \"maas-controller-bc7f7d7c4-b4w5d\" (UID: \"2fe32e67-b5ae-40d5-9b7f-530f984faea2\") " pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.319688 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.319601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:18.590609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.590584    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc7f7d7c4-b4w5d"]
Sep 15 12:38:18.592871 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:38:18.592835    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe32e67_b5ae_40d5_9b7f_530f984faea2.slice/crio-bb4f2763cb2011ae6a1bf22400d87e0ad941fa98deb716fc79b35bee5eabf463 WatchSource:0}: Error finding container bb4f2763cb2011ae6a1bf22400d87e0ad941fa98deb716fc79b35bee5eabf463: Status 404 returned error can't find the container with id bb4f2763cb2011ae6a1bf22400d87e0ad941fa98deb716fc79b35bee5eabf463
Sep 15 12:38:18.869058 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:18.868961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d" event={"ID":"2fe32e67-b5ae-40d5-9b7f-530f984faea2","Type":"ContainerStarted","Data":"bb4f2763cb2011ae6a1bf22400d87e0ad941fa98deb716fc79b35bee5eabf463"}
Sep 15 12:38:19.578302 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:19.578261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:19.875268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:19.875164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d" event={"ID":"2fe32e67-b5ae-40d5-9b7f-530f984faea2","Type":"ContainerStarted","Data":"a256e80f9f35a8d96b10b7dc1fc354eb4014d8f7c73559c98991ee24c5f2abc2"}
Sep 15 12:38:19.875268 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:19.875229    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:19.896236 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:19.896178    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d" podStartSLOduration=2.57519907 podStartE2EDuration="2.896160614s" totalImagesPullingTime="320.961544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:19.893208379 +0000 UTC m=+2180.589767956" watchObservedRunningTime="2026-09-15 12:38:19.896160614 +0000 UTC m=+2180.592720194"
Sep 15 12:38:30.882178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:30.882144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc7f7d7c4-b4w5d"
Sep 15 12:38:36.196114 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:36.196071    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:36.296780 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:38:36.296738    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:16.314163 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:16.314127    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:32.855260 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:32.855221    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"]
Sep 15 12:39:32.855859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:32.855498    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager" containerID="cri-o://94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b" gracePeriod=10
Sep 15 12:39:33.107440 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.107380    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:39:33.219884 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.219850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q97z\" (UniqueName: \"kubernetes.io/projected/d2341045-6356-4333-b9fb-e3a7adfdbc14-kube-api-access-8q97z\") pod \"d2341045-6356-4333-b9fb-e3a7adfdbc14\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") "
Sep 15 12:39:33.220096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.219929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2341045-6356-4333-b9fb-e3a7adfdbc14-extensions-socket-volume\") pod \"d2341045-6356-4333-b9fb-e3a7adfdbc14\" (UID: \"d2341045-6356-4333-b9fb-e3a7adfdbc14\") "
Sep 15 12:39:33.220369 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.220330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2341045-6356-4333-b9fb-e3a7adfdbc14-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2341045-6356-4333-b9fb-e3a7adfdbc14" (UID: "d2341045-6356-4333-b9fb-e3a7adfdbc14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:39:33.221956 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.221935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2341045-6356-4333-b9fb-e3a7adfdbc14-kube-api-access-8q97z" (OuterVolumeSpecName: "kube-api-access-8q97z") pod "d2341045-6356-4333-b9fb-e3a7adfdbc14" (UID: "d2341045-6356-4333-b9fb-e3a7adfdbc14"). InnerVolumeSpecName "kube-api-access-8q97z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:39:33.240973 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.240946    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerID="94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b" exitCode=0
Sep 15 12:39:33.241088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.240995    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"
Sep 15 12:39:33.241088 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.240997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" event={"ID":"d2341045-6356-4333-b9fb-e3a7adfdbc14","Type":"ContainerDied","Data":"94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b"}
Sep 15 12:39:33.241178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.241086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7" event={"ID":"d2341045-6356-4333-b9fb-e3a7adfdbc14","Type":"ContainerDied","Data":"cf059ea9434729394f871637b9f1531a87929015d15d1d891577d5c0940f18e1"}
Sep 15 12:39:33.241178 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.241126    2583 scope.go:117] "RemoveContainer" containerID="94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b"
Sep 15 12:39:33.250887 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.250869    2583 scope.go:117] "RemoveContainer" containerID="70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"
Sep 15 12:39:33.262661 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.262622    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"]
Sep 15 12:39:33.262741 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.262681    2583 scope.go:117] "RemoveContainer" containerID="94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b"
Sep 15 12:39:33.263008 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:39:33.262985    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b\": container with ID starting with 94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b not found: ID does not exist" containerID="94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b"
Sep 15 12:39:33.263096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.263015    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b"} err="failed to get container status \"94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b\": rpc error: code = NotFound desc = could not find container \"94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b\": container with ID starting with 94dd5485de343d9995b1146abc13eda6a052c530de4de8a93618d111c9699b0b not found: ID does not exist"
Sep 15 12:39:33.263096 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.263050    2583 scope.go:117] "RemoveContainer" containerID="70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"
Sep 15 12:39:33.263334 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:39:33.263316    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616\": container with ID starting with 70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616 not found: ID does not exist" containerID="70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"
Sep 15 12:39:33.263374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.263341    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616"} err="failed to get container status \"70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616\": rpc error: code = NotFound desc = could not find container \"70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616\": container with ID starting with 70e5eee8690fef249f1d78a167d4b833e22d8c133a3f012c738f4cc16e45c616 not found: ID does not exist"
Sep 15 12:39:33.266678 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.266656    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-88bt7"]
Sep 15 12:39:33.321112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.321057    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2341045-6356-4333-b9fb-e3a7adfdbc14-extensions-socket-volume\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:39:33.321112 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.321108    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q97z\" (UniqueName: \"kubernetes.io/projected/d2341045-6356-4333-b9fb-e3a7adfdbc14-kube-api-access-8q97z\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:39:33.807591 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:39:33.807560    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" path="/var/lib/kubelet/pods/d2341045-6356-4333-b9fb-e3a7adfdbc14/volumes"
Sep 15 12:40:33.524749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.524711    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"]
Sep 15 12:40:33.525270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.525159    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager"
Sep 15 12:40:33.525270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.525172    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager"
Sep 15 12:40:33.525270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.525186    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager"
Sep 15 12:40:33.525270 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.525192    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager"
Sep 15 12:40:33.525413 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.525290    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager"
Sep 15 12:40:33.525413 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.525301    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2341045-6356-4333-b9fb-e3a7adfdbc14" containerName="manager"
Sep 15 12:40:33.528445 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.528425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.532711 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.532687    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-49cwg\""
Sep 15 12:40:33.540475 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.540452    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"]
Sep 15 12:40:33.645646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.645599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ccm\" (UniqueName: \"kubernetes.io/projected/68bee70f-3b99-4443-869e-942d6a8063ad-kube-api-access-z2ccm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w9znw\" (UID: \"68bee70f-3b99-4443-869e-942d6a8063ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.645646 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.645650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68bee70f-3b99-4443-869e-942d6a8063ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w9znw\" (UID: \"68bee70f-3b99-4443-869e-942d6a8063ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.746572 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.746537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ccm\" (UniqueName: \"kubernetes.io/projected/68bee70f-3b99-4443-869e-942d6a8063ad-kube-api-access-z2ccm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w9znw\" (UID: \"68bee70f-3b99-4443-869e-942d6a8063ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.746739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.746576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68bee70f-3b99-4443-869e-942d6a8063ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w9znw\" (UID: \"68bee70f-3b99-4443-869e-942d6a8063ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.746953 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.746936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68bee70f-3b99-4443-869e-942d6a8063ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w9znw\" (UID: \"68bee70f-3b99-4443-869e-942d6a8063ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.755494 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.755470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ccm\" (UniqueName: \"kubernetes.io/projected/68bee70f-3b99-4443-869e-942d6a8063ad-kube-api-access-z2ccm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w9znw\" (UID: \"68bee70f-3b99-4443-869e-942d6a8063ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:33.838821 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:33.838788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:34.116279 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:34.116254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"]
Sep 15 12:40:34.121412 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:40:34.121378    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bee70f_3b99_4443_869e_942d6a8063ad.slice/crio-3913fb1bec8bd6fab680ed92daa5037c94c446186a9674b0bf4e899c76fe58ab WatchSource:0}: Error finding container 3913fb1bec8bd6fab680ed92daa5037c94c446186a9674b0bf4e899c76fe58ab: Status 404 returned error can't find the container with id 3913fb1bec8bd6fab680ed92daa5037c94c446186a9674b0bf4e899c76fe58ab
Sep 15 12:40:34.539491 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:34.539389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw" event={"ID":"68bee70f-3b99-4443-869e-942d6a8063ad","Type":"ContainerStarted","Data":"4c96729eb08f1a085d26a98001e206a3276a299bd66fd2911151246c00108c3a"}
Sep 15 12:40:34.539491 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:34.539448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw" event={"ID":"68bee70f-3b99-4443-869e-942d6a8063ad","Type":"ContainerStarted","Data":"3913fb1bec8bd6fab680ed92daa5037c94c446186a9674b0bf4e899c76fe58ab"}
Sep 15 12:40:34.539491 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:34.539476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:34.564563 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:34.564497    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw" podStartSLOduration=1.5644821530000002 podStartE2EDuration="1.564482153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:40:34.563867764 +0000 UTC m=+2315.260427354" watchObservedRunningTime="2026-09-15 12:40:34.564482153 +0000 UTC m=+2315.261041731"
Sep 15 12:40:45.545960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:45.545929    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w9znw"
Sep 15 12:40:56.267472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:40:56.267390    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:08.134141 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:41:08.134107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:37.105982 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:41:37.105927    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:59.956344 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:41:59.956316    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:41:59.963529 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:41:59.963498    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:42:04.491633 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:04.491597    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:05.334435 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:05.334374    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:25.470557 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.470521    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"]
Sep 15 12:42:25.474307 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.474284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.484560 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.484533    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"]
Sep 15 12:42:25.622827 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.622791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb822084-44eb-4a65-93a9-79117c576a76-openshift-service-ca\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.623005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.622837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb822084-44eb-4a65-93a9-79117c576a76-metrics-certs\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.623005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.622904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb822084-44eb-4a65-93a9-79117c576a76-maas-api-tls\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.623005 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.622950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmlj\" (UniqueName: \"kubernetes.io/projected/bb822084-44eb-4a65-93a9-79117c576a76-kube-api-access-mwmlj\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.723773 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.723691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb822084-44eb-4a65-93a9-79117c576a76-openshift-service-ca\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.723773 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.723735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb822084-44eb-4a65-93a9-79117c576a76-metrics-certs\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.723993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.723818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb822084-44eb-4a65-93a9-79117c576a76-maas-api-tls\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.723993 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.723943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmlj\" (UniqueName: \"kubernetes.io/projected/bb822084-44eb-4a65-93a9-79117c576a76-kube-api-access-mwmlj\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.724518 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.724482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb822084-44eb-4a65-93a9-79117c576a76-openshift-service-ca\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.726378 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.726352    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb822084-44eb-4a65-93a9-79117c576a76-maas-api-tls\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.726472 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.726405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb822084-44eb-4a65-93a9-79117c576a76-metrics-certs\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.731623 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.731603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmlj\" (UniqueName: \"kubernetes.io/projected/bb822084-44eb-4a65-93a9-79117c576a76-kube-api-access-mwmlj\") pod \"maas-api-5dd5d756d5-fqpnn\" (UID: \"bb822084-44eb-4a65-93a9-79117c576a76\") " pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:25.789688 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:25.789647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:26.068239 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:26.068206    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"]
Sep 15 12:42:26.071146 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:42:26.071122    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb822084_44eb_4a65_93a9_79117c576a76.slice/crio-24f22c51da1f6261ea5e251685ec5156f5693f5686509289c58c57794f518b16 WatchSource:0}: Error finding container 24f22c51da1f6261ea5e251685ec5156f5693f5686509289c58c57794f518b16: Status 404 returned error can't find the container with id 24f22c51da1f6261ea5e251685ec5156f5693f5686509289c58c57794f518b16
Sep 15 12:42:26.085147 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:26.085119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn" event={"ID":"bb822084-44eb-4a65-93a9-79117c576a76","Type":"ContainerStarted","Data":"24f22c51da1f6261ea5e251685ec5156f5693f5686509289c58c57794f518b16"}
Sep 15 12:42:27.092478 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:27.092440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn" event={"ID":"bb822084-44eb-4a65-93a9-79117c576a76","Type":"ContainerStarted","Data":"b0c09bb1f2aeeca197da75bd7f6f73ed8a3611ffb39be861bd9a67fa305445ad"}
Sep 15 12:42:27.092902 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:27.092561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:27.110393 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:27.110332    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn" podStartSLOduration=1.221897631 podStartE2EDuration="2.11031065s" totalImagesPullingTime="888.413019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:42:27.109250313 +0000 UTC m=+2427.805809892" watchObservedRunningTime="2026-09-15 12:42:27.11031065 +0000 UTC m=+2427.806870231"
Sep 15 12:42:33.103185 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:33.103150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5dd5d756d5-fqpnn"
Sep 15 12:42:41.441561 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:41.441523    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-865999984f-229wk"]
Sep 15 12:42:41.443952 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:41.441739    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" podUID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" containerName="maas-api" containerID="cri-o://80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773" gracePeriod=30
Sep 15 12:42:41.801361 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:41.801263    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" podUID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.46:8443/health\": dial tcp 10.132.0.46:8443: connect: connection refused"
Sep 15 12:42:42.699801 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.699776    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:42:42.879325 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.879230    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9xv2\" (UniqueName: \"kubernetes.io/projected/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-kube-api-access-g9xv2\") pod \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") "
Sep 15 12:42:42.879508 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.879360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-metrics-certs\") pod \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") "
Sep 15 12:42:42.879508 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.879436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls\") pod \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") "
Sep 15 12:42:42.879508 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.879463    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-openshift-service-ca\") pod \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\" (UID: \"fffb7b5b-85ff-4f02-919a-b49ca8a5b476\") "
Sep 15 12:42:42.879851 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.879819    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fffb7b5b-85ff-4f02-919a-b49ca8a5b476" (UID: "fffb7b5b-85ff-4f02-919a-b49ca8a5b476"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:42:42.881430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.881407    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fffb7b5b-85ff-4f02-919a-b49ca8a5b476" (UID: "fffb7b5b-85ff-4f02-919a-b49ca8a5b476"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:42:42.881532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.881486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-kube-api-access-g9xv2" (OuterVolumeSpecName: "kube-api-access-g9xv2") pod "fffb7b5b-85ff-4f02-919a-b49ca8a5b476" (UID: "fffb7b5b-85ff-4f02-919a-b49ca8a5b476"). InnerVolumeSpecName "kube-api-access-g9xv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:42:42.881532 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.881519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fffb7b5b-85ff-4f02-919a-b49ca8a5b476" (UID: "fffb7b5b-85ff-4f02-919a-b49ca8a5b476"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:42:42.980339 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.980301    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xv2\" (UniqueName: \"kubernetes.io/projected/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-kube-api-access-g9xv2\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:42:42.980543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.980350    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-metrics-certs\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:42:42.980543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.980378    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-maas-api-tls\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:42:42.980543 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:42.980394    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fffb7b5b-85ff-4f02-919a-b49ca8a5b476-openshift-service-ca\") on node \"ip-10-0-140-229.ec2.internal\" DevicePath \"\""
Sep 15 12:42:43.174392 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.174300    2583 generic.go:356] "Generic (PLEG): container finished" podID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" containerID="80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773" exitCode=0
Sep 15 12:42:43.174392 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.174348    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk"
Sep 15 12:42:43.174587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.174389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" event={"ID":"fffb7b5b-85ff-4f02-919a-b49ca8a5b476","Type":"ContainerDied","Data":"80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773"}
Sep 15 12:42:43.174587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.174441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-865999984f-229wk" event={"ID":"fffb7b5b-85ff-4f02-919a-b49ca8a5b476","Type":"ContainerDied","Data":"8244afef5c4cbe0f0507ad04bdce14508341c8eb432f478a506f3058217d5d20"}
Sep 15 12:42:43.174587 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.174464    2583 scope.go:117] "RemoveContainer" containerID="80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773"
Sep 15 12:42:43.184035 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.184008    2583 scope.go:117] "RemoveContainer" containerID="80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773"
Sep 15 12:42:43.184348 ip-10-0-140-229 kubenswrapper[2583]: E0915 12:42:43.184331    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773\": container with ID starting with 80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773 not found: ID does not exist" containerID="80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773"
Sep 15 12:42:43.184404 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.184358    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773"} err="failed to get container status \"80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773\": rpc error: code = NotFound desc = could not find container \"80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773\": container with ID starting with 80d3332d298f37a83e1d4978354127ae3c1922058edb96ad3c45bae0729ed773 not found: ID does not exist"
Sep 15 12:42:43.194835 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.194810    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-865999984f-229wk"]
Sep 15 12:42:43.198824 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.198802    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-865999984f-229wk"]
Sep 15 12:42:43.808707 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:43.808620    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" path="/var/lib/kubelet/pods/fffb7b5b-85ff-4f02-919a-b49ca8a5b476/volumes"
Sep 15 12:42:44.246949 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:44.246915    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:44.347346 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:42:44.347313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:05.013600 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:05.013569    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-45ggs_cc2173ee-9149-4649-810b-440af589aa66/manager/0.log"
Sep 15 12:43:05.256139 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:05.256100    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc7f7d7c4-b4w5d_2fe32e67-b5ae-40d5-9b7f-530f984faea2/manager/0.log"
Sep 15 12:43:05.730396 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:05.730368    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-p9wwk_25757abf-ba99-4912-9785-d946796d63a4/copy-manifests/0.log"
Sep 15 12:43:05.736745 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:05.736725    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-p9wwk_25757abf-ba99-4912-9785-d946796d63a4/manager/0.log"
Sep 15 12:43:06.193124 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:06.193087    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5dd5d756d5-fqpnn_bb822084-44eb-4a65-93a9-79117c576a76/maas-api/0.log"
Sep 15 12:43:06.624210 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:06.624182    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lmpjp_f93cf73b-ec28-4397-8eb5-a71b267064e9/postgres/0.log"
Sep 15 12:43:07.491354 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:07.491318    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-lmbsz_569e7a31-2a8e-4364-b182-146adc5d1c3f/manager/0.log"
Sep 15 12:43:07.612191 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:07.612159    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-b88lm_e6de30c0-901b-4880-83a9-70ca1ad48718/manager/0.log"
Sep 15 12:43:07.840833 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:07.840793    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-w9znw_68bee70f-3b99-4443-869e-942d6a8063ad/manager/0.log"
Sep 15 12:43:08.074050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:08.074002    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-cs744_50fd61a8-ba57-4d44-9265-0da863990d31/manager/0.log"
Sep 15 12:43:08.757442 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:08.757415    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-mk969_a36df0a6-b85d-417f-97bf-e496036f2e1e/kube-auth-proxy/0.log"
Sep 15 12:43:08.991216 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:08.991181    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-26h95_3dc0d539-2e6f-4b87-8ecb-d01c327ad983/payload-pre-processing/0.log"
Sep 15 12:43:09.223624 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:09.223587    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847cffd544-fxqfj_ef818076-7989-462f-b9f6-92a7cc8352d8/router/0.log"
Sep 15 12:43:09.681504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:09.681466    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-rrd5t_af111aac-ecbe-4b67-a9f0-fb15a1b2c001/main/0.log"
Sep 15 12:43:15.700622 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.700585    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n4f4n/must-gather-h5g4x"]
Sep 15 12:43:15.701070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.700984    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" containerName="maas-api"
Sep 15 12:43:15.701070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.700995    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" containerName="maas-api"
Sep 15 12:43:15.701070 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.701065    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb7b5b-85ff-4f02-919a-b49ca8a5b476" containerName="maas-api"
Sep 15 12:43:15.704229 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.704205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:15.706406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.706382    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4f4n\"/\"kube-root-ca.crt\""
Sep 15 12:43:15.706522 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.706412    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n4f4n\"/\"default-dockercfg-27rxc\""
Sep 15 12:43:15.707259 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.707239    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4f4n\"/\"openshift-service-ca.crt\""
Sep 15 12:43:15.708429 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.708401    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:15.713772 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.713746    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4f4n/must-gather-h5g4x"]
Sep 15 12:43:15.767686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.767649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/021044fa-0aa8-4671-b991-52c54f9d00fa-must-gather-output\") pod \"must-gather-h5g4x\" (UID: \"021044fa-0aa8-4671-b991-52c54f9d00fa\") " pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:15.767686 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.767703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbq4\" (UniqueName: \"kubernetes.io/projected/021044fa-0aa8-4671-b991-52c54f9d00fa-kube-api-access-znbq4\") pod \"must-gather-h5g4x\" (UID: \"021044fa-0aa8-4671-b991-52c54f9d00fa\") " pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:15.774173 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.774133    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:15.868644 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.868609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/021044fa-0aa8-4671-b991-52c54f9d00fa-must-gather-output\") pod \"must-gather-h5g4x\" (UID: \"021044fa-0aa8-4671-b991-52c54f9d00fa\") " pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:15.868844 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.868675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbq4\" (UniqueName: \"kubernetes.io/projected/021044fa-0aa8-4671-b991-52c54f9d00fa-kube-api-access-znbq4\") pod \"must-gather-h5g4x\" (UID: \"021044fa-0aa8-4671-b991-52c54f9d00fa\") " pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:15.868983 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.868964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/021044fa-0aa8-4671-b991-52c54f9d00fa-must-gather-output\") pod \"must-gather-h5g4x\" (UID: \"021044fa-0aa8-4671-b991-52c54f9d00fa\") " pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:15.876345 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:15.876318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbq4\" (UniqueName: \"kubernetes.io/projected/021044fa-0aa8-4671-b991-52c54f9d00fa-kube-api-access-znbq4\") pod \"must-gather-h5g4x\" (UID: \"021044fa-0aa8-4671-b991-52c54f9d00fa\") " pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:16.014050 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:16.013931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4f4n/must-gather-h5g4x"
Sep 15 12:43:16.287152 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:16.287127    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4f4n/must-gather-h5g4x"]
Sep 15 12:43:16.289849 ip-10-0-140-229 kubenswrapper[2583]: W0915 12:43:16.289824    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod021044fa_0aa8_4671_b991_52c54f9d00fa.slice/crio-adb3d699c30111a53cc3d67b0b545d513f116e3da3626083b240e88efb2b5762 WatchSource:0}: Error finding container adb3d699c30111a53cc3d67b0b545d513f116e3da3626083b240e88efb2b5762: Status 404 returned error can't find the container with id adb3d699c30111a53cc3d67b0b545d513f116e3da3626083b240e88efb2b5762
Sep 15 12:43:16.291647 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:16.291625    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:43:16.343612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:16.343579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/must-gather-h5g4x" event={"ID":"021044fa-0aa8-4671-b991-52c54f9d00fa","Type":"ContainerStarted","Data":"adb3d699c30111a53cc3d67b0b545d513f116e3da3626083b240e88efb2b5762"}
Sep 15 12:43:17.353385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:17.353295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/must-gather-h5g4x" event={"ID":"021044fa-0aa8-4671-b991-52c54f9d00fa","Type":"ContainerStarted","Data":"fa2924ead0c4e493a286e9828fba6b44b06d01d1b9b3dea6b8c18bdce2b7440d"}
Sep 15 12:43:17.353385 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:17.353339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/must-gather-h5g4x" event={"ID":"021044fa-0aa8-4671-b991-52c54f9d00fa","Type":"ContainerStarted","Data":"0285d622f9d276a0de0a9dbe9f3ee75c3dd96c3cdee022e91c05882847bd9b47"}
Sep 15 12:43:18.588286 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:18.588246    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-66f9b_cec4d24d-6cbf-485b-9754-3ab98fce5a6f/global-pull-secret-syncer/0.log"
Sep 15 12:43:18.727969 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:18.727935    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bxmpx_ca52c69a-01eb-448e-86c0-7a8018d39ad8/konnectivity-agent/0.log"
Sep 15 12:43:18.866374 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:18.866277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-229.ec2.internal_23cb9ee72bd5fd0349ade4265f4e6c00/haproxy/0.log"
Sep 15 12:43:24.053760 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:24.053683    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-lmbsz_569e7a31-2a8e-4364-b182-146adc5d1c3f/manager/0.log"
Sep 15 12:43:24.081273 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:24.081232    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-b88lm_e6de30c0-901b-4880-83a9-70ca1ad48718/manager/0.log"
Sep 15 12:43:24.164397 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:24.164354    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-w9znw_68bee70f-3b99-4443-869e-942d6a8063ad/manager/0.log"
Sep 15 12:43:24.255338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:24.255240    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-cs744_50fd61a8-ba57-4d44-9265-0da863990d31/manager/0.log"
Sep 15 12:43:25.794749 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:25.794710    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-b5cnn_da6c3092-9ef2-4a81-9a90-0deecff7c4bc/cluster-monitoring-operator/0.log"
Sep 15 12:43:25.971046 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:25.967869    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j45ww_47328fc0-2033-4a89-8479-01152dfcf1e8/node-exporter/0.log"
Sep 15 12:43:26.009960 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:26.009917    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j45ww_47328fc0-2033-4a89-8479-01152dfcf1e8/kube-rbac-proxy/0.log"
Sep 15 12:43:26.043657 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:26.043627    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j45ww_47328fc0-2033-4a89-8479-01152dfcf1e8/init-textfile/0.log"
Sep 15 12:43:26.565363 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:26.565333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-cw7jn_4017423e-1b35-4c92-ae30-65a8c32fc94b/prometheus-operator-admission-webhook/0.log"
Sep 15 12:43:28.106612 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:28.106583    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-lfzm7_31a288ab-9731-41ec-8b6d-cc37c790fe6f/networking-console-plugin/0.log"
Sep 15 12:43:28.675755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:28.675725    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/1.log"
Sep 15 12:43:28.682609 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:28.682568    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8zw5m_c5570725-5835-48e5-8a10-599f37cffed7/console-operator/2.log"
Sep 15 12:43:29.002409 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.002295    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n4f4n/must-gather-h5g4x" podStartSLOduration=13.22499812 podStartE2EDuration="14.002272122s" totalImagesPullingTime="777.274002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:43:17.368576072 +0000 UTC m=+2478.065135651" watchObservedRunningTime="2026-09-15 12:43:29.002272122 +0000 UTC m=+2489.698831699"
Sep 15 12:43:29.003459 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.003422    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"]
Sep 15 12:43:29.009572 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.009548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.016350 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.016322    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"]
Sep 15 12:43:29.125951 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.125913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-lib-modules\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.126430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.125968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-sys\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.126430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.126018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlx8\" (UniqueName: \"kubernetes.io/projected/3b6b067e-4ee6-4dd8-91bd-82463724151c-kube-api-access-zvlx8\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.126430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.126118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-podres\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.126430 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.126157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-proc\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.227568 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-podres\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.227755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-proc\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.227755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-lib-modules\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.227755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227706    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-podres\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.227755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-proc\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.227755 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-sys\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.228063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-sys\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.228063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlx8\" (UniqueName: \"kubernetes.io/projected/3b6b067e-4ee6-4dd8-91bd-82463724151c-kube-api-access-zvlx8\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.228063 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.227860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b6b067e-4ee6-4dd8-91bd-82463724151c-lib-modules\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.236927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.236892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlx8\" (UniqueName: \"kubernetes.io/projected/3b6b067e-4ee6-4dd8-91bd-82463724151c-kube-api-access-zvlx8\") pod \"perf-node-gather-daemonset-gjdtr\" (UID: \"3b6b067e-4ee6-4dd8-91bd-82463724151c\") " pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.296014 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.295907    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-xmttb_d93ac03f-b94b-4d6f-b8e4-857222a82144/download-server/0.log"
Sep 15 12:43:29.324098 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.324011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:29.623354 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:29.623313    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"]
Sep 15 12:43:30.017635 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:30.017559    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-x664q_40dc7c0c-4616-4f6e-bde1-568ad50936af/volume-data-source-validator/0.log"
Sep 15 12:43:30.456338 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:30.456289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr" event={"ID":"3b6b067e-4ee6-4dd8-91bd-82463724151c","Type":"ContainerStarted","Data":"173e5c2042c83bbaa5333f2f42101f7bd00fc1b03268936c2c2eafad72d1ff3b"}
Sep 15 12:43:30.456792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:30.456366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr" event={"ID":"3b6b067e-4ee6-4dd8-91bd-82463724151c","Type":"ContainerStarted","Data":"5252565bfee7dfe809f18dff295880cc0259684edafd5eea5cfab0f665f2faf4"}
Sep 15 12:43:30.456792 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:30.456527    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:30.472739 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:30.472683    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr" podStartSLOduration=2.472667377 podStartE2EDuration="2.472667377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:43:30.470450034 +0000 UTC m=+2491.167009632" watchObservedRunningTime="2026-09-15 12:43:30.472667377 +0000 UTC m=+2491.169226946"
Sep 15 12:43:31.177636 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:31.177586    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8ckhc_3b813070-33ae-426d-ad4a-1fba43dfc263/dns/0.log"
Sep 15 12:43:31.196376 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:31.196343    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8ckhc_3b813070-33ae-426d-ad4a-1fba43dfc263/kube-rbac-proxy/0.log"
Sep 15 12:43:31.306626 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:31.306595    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xw4k7_144046aa-3d7c-4925-8259-da5480d92d3e/dns-node-resolver/0.log"
Sep 15 12:43:31.874016 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:31.873977    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f79s4_ea5ba13e-d405-4fb7-9030-811b949df8d8/node-ca/0.log"
Sep 15 12:43:33.677841 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:33.677789    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-mk969_a36df0a6-b85d-417f-97bf-e496036f2e1e/kube-auth-proxy/0.log"
Sep 15 12:43:33.745176 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:33.745146    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-26h95_3dc0d539-2e6f-4b87-8ecb-d01c327ad983/payload-pre-processing/0.log"
Sep 15 12:43:33.815389 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:33.815349    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847cffd544-fxqfj_ef818076-7989-462f-b9f6-92a7cc8352d8/router/0.log"
Sep 15 12:43:34.728173 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:34.728147    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zcjwh_81040b81-30da-48ef-963b-d42e89a16b58/serve-healthcheck-canary/0.log"
Sep 15 12:43:35.194534 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:35.194497    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-s5npn_0b404c4b-5d10-407c-9193-4eb1fe994a97/insights-operator/0.log"
Sep 15 12:43:35.196859 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:35.196836    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-s5npn_0b404c4b-5d10-407c-9193-4eb1fe994a97/insights-operator/1.log"
Sep 15 12:43:35.312151 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:35.312121    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4w7s9_ef233600-bc63-4e81-9078-b87df9e76dbf/kube-rbac-proxy/0.log"
Sep 15 12:43:35.333271 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:35.333238    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4w7s9_ef233600-bc63-4e81-9078-b87df9e76dbf/exporter/0.log"
Sep 15 12:43:35.354497 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:35.354465    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4w7s9_ef233600-bc63-4e81-9078-b87df9e76dbf/extractor/0.log"
Sep 15 12:43:36.476675 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:36.476645    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n4f4n/perf-node-gather-daemonset-gjdtr"
Sep 15 12:43:37.561400 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:37.561372    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-45ggs_cc2173ee-9149-4649-810b-440af589aa66/manager/0.log"
Sep 15 12:43:37.887505 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:37.887425    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc7f7d7c4-b4w5d_2fe32e67-b5ae-40d5-9b7f-530f984faea2/manager/0.log"
Sep 15 12:43:38.228897 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:38.228769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-p9wwk_25757abf-ba99-4912-9785-d946796d63a4/manager/0.log"
Sep 15 12:43:38.248528 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:38.248496    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b747b9585-p9wwk_25757abf-ba99-4912-9785-d946796d63a4/copy-manifests/0.log"
Sep 15 12:43:40.282319 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:40.282286    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-mk969_a36df0a6-b85d-417f-97bf-e496036f2e1e/kube-auth-proxy/0.log"
Sep 15 12:43:40.334990 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:40.334950    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-26h95_3dc0d539-2e6f-4b87-8ecb-d01c327ad983/payload-pre-processing/0.log"
Sep 15 12:43:40.397377 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:40.397352    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847cffd544-fxqfj_ef818076-7989-462f-b9f6-92a7cc8352d8/router/0.log"
Sep 15 12:43:43.059551 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:43.059519    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cf78458b7-mk969_a36df0a6-b85d-417f-97bf-e496036f2e1e/kube-auth-proxy/0.log"
Sep 15 12:43:43.119793 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:43.119761    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-26h95_3dc0d539-2e6f-4b87-8ecb-d01c327ad983/payload-pre-processing/0.log"
Sep 15 12:43:43.170446 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:43.170406    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847cffd544-fxqfj_ef818076-7989-462f-b9f6-92a7cc8352d8/router/0.log"
Sep 15 12:43:44.631314 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:44.631284    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ph8vj_eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5/migrator/0.log"
Sep 15 12:43:44.652785 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:44.652750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-ph8vj_eb0a7ed2-ffa3-48d0-83f3-7601b4e458c5/graceful-termination/0.log"
Sep 15 12:43:45.022406 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:45.022288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-ml4vj_47fdb44b-602b-4b18-b9f7-e1a11db46547/kube-storage-version-migrator-operator/1.log"
Sep 15 12:43:45.023504 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:45.023476    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-ml4vj_47fdb44b-602b-4b18-b9f7-e1a11db46547/kube-storage-version-migrator-operator/0.log"
Sep 15 12:43:46.014226 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.014190    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/kube-multus-additional-cni-plugins/0.log"
Sep 15 12:43:46.034962 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.034938    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/egress-router-binary-copy/0.log"
Sep 15 12:43:46.054540 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.054478    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/cni-plugins/0.log"
Sep 15 12:43:46.073563 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.073535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/bond-cni-plugin/0.log"
Sep 15 12:43:46.092857 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.092829    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/routeoverride-cni/0.log"
Sep 15 12:43:46.111886 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.111862    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/whereabouts-cni-bincopy/0.log"
Sep 15 12:43:46.134817 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.134792    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9qdwb_15c02746-890f-4bfa-9bbe-c20980af3d45/whereabouts-cni/0.log"
Sep 15 12:43:46.538718 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.538688    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lct5p_f91cc570-5cdc-41a7-8076-28d2370630f7/kube-multus/0.log"
Sep 15 12:43:46.603702 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.603671    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g2n6z_59f50a12-82fe-48ec-8b9f-26aa7c1133d4/network-metrics-daemon/0.log"
Sep 15 12:43:46.618927 ip-10-0-140-229 kubenswrapper[2583]: I0915 12:43:46.618899    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g2n6z_59f50a12-82fe-48ec-8b9f-26aa7c1133d4/kube-rbac-proxy/0.log"