Sep 15 09:06:47.420125 ip-10-0-134-28 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 09:06:47.835047 ip-10-0-134-28 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 09:06:47.835047 ip-10-0-134-28 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 09:06:47.835047 ip-10-0-134-28 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 09:06:47.835047 ip-10-0-134-28 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 09:06:47.835047 ip-10-0-134-28 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 09:06:47.837111 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.837015    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842224    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842259    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842264    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842268    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842272    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842275    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842278    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842280    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:06:47.842263 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842283    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842286    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842289    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842292    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842294    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842297    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842300    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842303    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842305    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842308    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842310    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842313    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842315    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842318    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842321    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842323    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842326    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842328    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842331    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:06:47.842574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842333    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842337    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842342    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842351    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842354    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842356    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842359    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842361    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842364    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842367    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842370    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842373    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842375    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842378    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842380    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842385    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842387    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842390    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842393    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842395    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:06:47.843042 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842398    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842401    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842403    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842406    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842408    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842411    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842414    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842416    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842419    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842421    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842425    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842431    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842434    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842437    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842440    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842443    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842446    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842448    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842451    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842453    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:06:47.843569 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842456    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842458    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842461    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842463    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842466    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842468    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842471    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842475    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842478    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842480    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842484    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842486    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842489    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842492    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842496    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842499    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842503    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842506    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842509    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842512    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:06:47.844065 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842515    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842518    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842521    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842523    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842526    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842528    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842531    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842534    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842537    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842539    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842542    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.842544    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843254    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843260    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843263    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843266    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843269    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843271    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843274    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843277    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:06:47.844549 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843280    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843282    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843285    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843288    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843292    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843296    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843299    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843301    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843304    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843308    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843310    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843313    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843316    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843318    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843321    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843323    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843326    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843328    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843331    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:06:47.845069 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843333    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843336    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843338    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843341    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843344    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843346    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843349    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843351    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843354    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843356    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843359    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843362    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843365    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843368    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843370    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843373    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843376    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843378    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843381    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843384    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843386    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:06:47.845539 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843389    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843391    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843394    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843397    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843400    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843402    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843405    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843407    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843410    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843412    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843415    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843417    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843420    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843422    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843425    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843428    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843431    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843434    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843436    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843439    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:06:47.846061 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843442    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843444    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843448    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843450    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843453    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843455    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843457    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843460    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843462    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843465    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843469    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843472    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843476    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843479    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843482    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843486    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843489    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843491    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843494    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843496    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:06:47.846544 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843499    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843501    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843505    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843508    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843511    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843517    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843519    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843522    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843525    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843527    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843530    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843930    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843936    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843939    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843942    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843945    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843948    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843951    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843953    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:06:47.847039 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843956    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843958    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843961    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843965    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843969    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843972    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843975    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843978    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843982    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843984    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843987    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843990    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843992    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843995    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.843997    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844000    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844002    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844005    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844009    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:06:47.847511 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844012    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844014    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844017    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844019    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844022    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844024    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844027    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844029    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844032    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844035    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844038    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844040    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844043    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844045    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844049    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844051    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844054    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844056    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844059    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844061    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:06:47.847976 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844064    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844066    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844070    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844072    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844075    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844078    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844080    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844083    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844085    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844089    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844092    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844096    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844099    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844102    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844104    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844107    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844109    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844112    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844114    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:06:47.848455 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844117    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844120    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844123    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844125    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844128    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844130    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844133    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844138    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844141    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844144    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844147    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844150    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844152    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844155    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844157    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844160    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844163    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844166    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844168    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844171    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:06:47.849055 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844173    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844176    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844179    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844181    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844185    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844189    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844192    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844195    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844197    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844200    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844203    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844205    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844208    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844286    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844331    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844338    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844342    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844346    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844349    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844354    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844358    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 09:06:47.849545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844361    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844364    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844368    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844371    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844374    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844377    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844380    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844383    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844387    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844391    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844395    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844398    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844402    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844405    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844408    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844414    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844419    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844422    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844425    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844429    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844431    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844435    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844438    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844441    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 09:06:47.850072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844445    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844448    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844451    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844454    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844457    2571 flags.go:64] FLAG: --enable-server="true"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844460    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844464    2571 flags.go:64] FLAG: --event-burst="100"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844467    2571 flags.go:64] FLAG: --event-qps="50"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844470    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844473    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844476    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844480    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844482    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844485    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844488    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844491    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844494    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844498    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844501    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844504    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844507    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844509    2571 flags.go:64] FLAG: --feature-gates=""
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844513    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844517    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844521    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 09:06:47.850686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844525    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844528    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844531    2571 flags.go:64] FLAG: --help="false"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844533    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-134-28.ec2.internal"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844537    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844540    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844543    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844546    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844550    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844553    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844556    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844559    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844562    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844565    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844568    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844571    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844574    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844577    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844580    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844582    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844585    2571 flags.go:64] FLAG: --lock-file=""
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844588    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844591    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844594    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 09:06:47.851377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844599    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844603    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844606    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844609    2571 flags.go:64] FLAG: --logging-format="text"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844618    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844623    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844627    2571 flags.go:64] FLAG: --manifest-url=""
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844632    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844636    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844639    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844643    2571 flags.go:64] FLAG: --max-pods="110"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844646    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844650    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844652    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844655    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844658    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844661    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844664    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844672    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844675    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844678    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844681    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844684    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 09:06:47.851974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844689    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844692    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844695    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844698    2571 flags.go:64] FLAG: --port="10250"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844701    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844704    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0faa75bd588a1921a"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844707    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844710    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844713    2571 flags.go:64] FLAG: --register-node="true"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844716    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844719    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844723    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844726    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844729    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844732    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844735    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844741    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844744    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844747    2571 flags.go:64] FLAG: --runonce="false"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844749    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844753    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844756    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844759    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844761    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844764    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844768    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 09:06:47.852554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844771    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844774    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844777    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844780    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844783    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844786    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844789    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844792    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844798    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844800    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844803    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844810    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844813    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844816    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844819    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844821    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844839    2571 flags.go:64] FLAG: --v="2"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844843    2571 flags.go:64] FLAG: --version="false"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844848    2571 flags.go:64] FLAG: --vmodule=""
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844852    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.844855    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844972    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844979    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844982    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:06:47.853223 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844985    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844988    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844990    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844993    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844996    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.844998    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845001    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845004    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845007    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845010    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845012    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845029    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845034    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845037    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845040    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845044    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845047    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845050    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845053    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:06:47.853868 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845055    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845058    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845061    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845063    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845066    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845068    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845071    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845074    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845079    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845082    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845085    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845092    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845095    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845097    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845100    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845103    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845105    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845108    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845111    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:06:47.854338 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845113    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845116    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845118    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845122    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845125    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845128    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845131    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845134    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845136    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845139    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845141    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845144    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845146    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845149    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845151    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845154    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845157    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845159    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845162    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845165    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:06:47.854811 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845167    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845170    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845173    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845175    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845180    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845183    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845185    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845188    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845191    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845194    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845196    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845199    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845202    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845205    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845207    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845210    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845212    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845215    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845217    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845220    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:06:47.855376 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845222    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845225    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845227    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845230    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845233    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845235    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845238    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845241    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845243    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845246    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845248    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845251    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845254    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845256    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845258    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845262    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845267    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:06:47.855887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.845269    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.845274    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.852413    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.852430    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.853241    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.853249    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853306    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853311    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853316    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853319    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853323    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853325    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853328    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853331    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:06:47.856324 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853335    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853340    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853343    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853346    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853349    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853352    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853356    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853359    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853361    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853364    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853367    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853370    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853373    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853376    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853379    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853381    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853384    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853387    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853390    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:06:47.856696 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853392    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853396    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853399    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853402    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853405    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853407    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853410    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853412    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853415    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853418    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853420    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853423    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853426    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853428    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853431    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853433    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853436    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853439    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853442    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:06:47.857261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853444    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853447    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853449    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853452    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853455    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853457    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853460    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853463    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853467    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853471    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853474    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853477    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853479    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853482    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853485    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853488    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853491    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853493    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853496    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853498    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:06:47.857768 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853501    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853504    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853506    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853509    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853511    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853514    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853516    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853519    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853521    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853524    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853526    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853529    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853531    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853534    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853537    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853539    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853542    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853544    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853547    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853549    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:06:47.858307 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853552    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853554    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853557    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853560    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853563    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853565    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853568    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853571    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853574    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853577    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853579    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853582    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853585    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.853590    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853710    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 09:06:47.858805 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853715    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853718    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853721    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853724    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853726    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853729    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853732    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853734    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853737    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853740    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853744    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853747    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853751    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853754    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853757    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853760    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853763    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853766    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853768    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 09:06:47.859239 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853772    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853774    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853777    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853780    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853783    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853786    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853788    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853791    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853794    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853797    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853799    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853802    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853804    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853807    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853809    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853812    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853814    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853817    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853819    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853822    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 09:06:47.859758 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853843    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853846    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853849    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853851    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853854    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853857    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853860    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853863    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853865    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853868    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853870    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853874    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853877    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853880    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853883    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853887    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853890    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853892    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853895    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853897    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 09:06:47.860296 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853900    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853904    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853907    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853910    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853912    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853915    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853917    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853920    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853922    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853925    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853927    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853930    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853933    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853935    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853938    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853941    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853943    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853946    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853948    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 09:06:47.860982 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853952    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853955    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853958    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853960    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853963    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853966    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853969    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853972    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853975    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853977    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853980    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853982    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853985    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853987    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853990    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853993    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853996    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.853998    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.854001    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 09:06:47.861449 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:47.854004    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.854009    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.854122    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.856001    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.856986    2571 server.go:1019] "Starting client certificate rotation"
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.857101    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.857161    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.859552    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 09:06:47.862080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.860701    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 09:06:47.876398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.876373    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 15 09:06:47.880214 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.880194    2571 log.go:25] "Validated CRI v1 image API"
Sep 15 09:06:47.881057 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.881041    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 09:06:47.884426 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.884405    2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7df1af30-ad00-4822-abdc-c25eb2cdfd9e:/dev/nvme0n1p4 a9ab9d16-68f6-4d0e-b4a7-555f35894eb5:/dev/nvme0n1p3]
Sep 15 09:06:47.884498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.884426    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 09:06:47.887422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.887404    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 09:06:47.890042 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.889925    2571 manager.go:216] Machine: {Timestamp:2026-09-15 09:06:47.888536792 +0000 UTC m=+0.347826324 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121603 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2adb9b7a832096fbb842147dd61930 SystemUUID:ec2adb9b-7a83-2096-fbb8-42147dd61930 BootID:b68e40e6-7f16-440c-a194-1f615e6da0b4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:58:32:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:58:32:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:19:8e:19:dc:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 09:06:47.890042 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.890035    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 09:06:47.890165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.890130    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 09:06:47.893685 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.893653    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 09:06:47.893863 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.893688    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 09:06:47.894524 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.894513    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 09:06:47.894555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.894525    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 09:06:47.894555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.894533    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 09:06:47.895320 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.895308    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 09:06:47.896151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.896141    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 09:06:47.899978 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.899964    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 09:06:47.900095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900086    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 09:06:47.900140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900116    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 09:06:47.900140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900126    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 09:06:47.900140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900141    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 09:06:47.900232 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900153    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 15 09:06:47.900232 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900162    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 09:06:47.900528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900513    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 09:06:47.900594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.900533    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 09:06:47.902894 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.902872    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 09:06:47.906689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.906673    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 09:06:47.906744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.906705    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 09:06:47.908442 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908407    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908449    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908461    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908470    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908492    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908500    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908521    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 09:06:47.908545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908530    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 09:06:47.908877 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908554    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 09:06:47.908877 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908564    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 09:06:47.908877 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908578    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 09:06:47.908877 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.908589    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 09:06:47.909566 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.909540    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 09:06:47.909661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.909598    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 09:06:47.914398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.914369    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 09:06:47.914885 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.914860    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 09:06:47.914885 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.914859    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 09:06:47.915328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.915314    2571 server.go:1268] "Started kubelet"
Sep 15 09:06:47.915433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.915391    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 09:06:47.915484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.915453    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 09:06:47.915534 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.915394    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 09:06:47.916248 ip-10-0-134-28 systemd[1]: Started Kubernetes Kubelet.
Sep 15 09:06:47.916753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.916735    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 09:06:47.917840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.917813    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 09:06:47.920692 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.920631    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 09:06:47.921145 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.920109    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18d5734e7e6bda02  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-09-15 09:06:47.915280898 +0000 UTC m=+0.374570485,LastTimestamp:2026-09-15 09:06:47.915280898 +0000 UTC m=+0.374570485,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Sep 15 09:06:47.922351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.922333    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 09:06:47.922351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.922345    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 09:06:47.922479 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.922411    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.922922    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.922946    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923039    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923081    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923090    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923307    2571 factory.go:55] Registering systemd factory
Sep 15 09:06:47.923502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923349    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 15 09:06:47.923893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923655    2571 factory.go:153] Registering CRI-O factory
Sep 15 09:06:47.923893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923673    2571 factory.go:223] Registration of the crio container factory successfully
Sep 15 09:06:47.923893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923735    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 09:06:47.923893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923756    2571 factory.go:103] Registering Raw factory
Sep 15 09:06:47.923893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.923773    2571 manager.go:1196] Started watching for new ooms in manager
Sep 15 09:06:47.924463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.924446    2571 manager.go:349] Starting recovery of all containers
Sep 15 09:06:47.930278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.930247    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 09:06:47.930551 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.930509    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 09:06:47.930668 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.930638    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 09:06:47.936224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.936174    2571 manager.go:354] Recovery completed
Sep 15 09:06:47.941788 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.941770    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:47.942223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.942199    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-grcr4"
Sep 15 09:06:47.943925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.943909    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:47.943994 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.943939    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:47.943994 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.943956    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:47.944474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.944459    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 09:06:47.944474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.944472    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 09:06:47.944588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.944486    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 09:06:47.944588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.944501    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:47.945418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.945400    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:47.945494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.945433    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:47.945494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.945448    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:47.945639 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.945396    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18d5734e8020ea8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-09-15 09:06:47.943924364 +0000 UTC m=+0.403213896,LastTimestamp:2026-09-15 09:06:47.943924364 +0000 UTC m=+0.403213896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Sep 15 09:06:47.946716 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.946704    2571 policy_none.go:49] "None policy: Start"
Sep 15 09:06:47.946756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.946720    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 09:06:47.946756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.946731    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 09:06:47.947746 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.947731    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-grcr4"
Sep 15 09:06:47.948755 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.948742    2571 policy_none.go:47] "Start"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.991512    2571 manager.go:337] "Starting Device Plugin manager"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.991560    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.991572    2571 server.go:87] "Starting device plugin registration server"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.991901    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.991914    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.992008    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.992093    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:47.992102    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.992727    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 09:06:48.004169 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:47.992774    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.048476 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.048443    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 09:06:48.048634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.048502    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 09:06:48.048634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.048530    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 09:06:48.048634 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.048574    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 09:06:48.051221 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.051193    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:06:48.092228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.092158    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:48.093173 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.093158    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:48.093249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.093185    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:48.093249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.093194    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:48.093249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.093216    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.100660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.100641    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.100707 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.100666    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-28.ec2.internal\": node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.128599 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.128423    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.149567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.149526    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"]
Sep 15 09:06:48.149704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.149608    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:48.151212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.151190    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:48.151319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.151220    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:48.151319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.151232    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:48.153418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.153405    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:48.153572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.153558    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.153619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.153595    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:48.154185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.154162    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:48.154185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.154183    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:48.154316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.154195    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:48.154316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.154230    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:48.154316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.154249    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:48.154316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.154264    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:48.156316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.156302    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.156399 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.156324    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 09:06:48.157071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.157048    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 09:06:48.157071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.157077    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 09:06:48.157197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.157092    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Sep 15 09:06:48.172413 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.172388    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-28.ec2.internal\" not found" node="ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.176415 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.176399    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-28.ec2.internal\" not found" node="ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.229276 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.229244    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.324564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.324525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3d0ab2acc9699061119b74368bceceb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"c3d0ab2acc9699061119b74368bceceb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.324564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.324566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d0ab2acc9699061119b74368bceceb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"c3d0ab2acc9699061119b74368bceceb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.324747 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.324598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff78554c81b115a40bbe63702b1bfba3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-28.ec2.internal\" (UID: \"ff78554c81b115a40bbe63702b1bfba3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.329571 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.329553    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.425580 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.425547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3d0ab2acc9699061119b74368bceceb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"c3d0ab2acc9699061119b74368bceceb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.425680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.425585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d0ab2acc9699061119b74368bceceb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"c3d0ab2acc9699061119b74368bceceb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.425680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.425616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff78554c81b115a40bbe63702b1bfba3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-28.ec2.internal\" (UID: \"ff78554c81b115a40bbe63702b1bfba3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.425754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.425663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3d0ab2acc9699061119b74368bceceb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"c3d0ab2acc9699061119b74368bceceb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.425754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.425690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff78554c81b115a40bbe63702b1bfba3-config\") pod \"kube-apiserver-proxy-ip-10-0-134-28.ec2.internal\" (UID: \"ff78554c81b115a40bbe63702b1bfba3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.425754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.425669    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3d0ab2acc9699061119b74368bceceb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"c3d0ab2acc9699061119b74368bceceb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.429909 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.429879    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.477706 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.477662    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.480025 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.480005    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:48.530025 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.529990    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.631058 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.630987    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.731737 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.731709    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.832048 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.832018    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.855987 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.855959    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:06:48.856972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.856956    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 09:06:48.857115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.857086    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 09:06:48.932201 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:48.932113    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:48.951475 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.951430    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 09:01:48 +0000 UTC" deadline="2028-05-19 06:35:47.601565366 +0000 UTC"
Sep 15 09:06:48.951475 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:48.951469    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14685h28m58.650100596s"
Sep 15 09:06:49.033321 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:49.033282    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:49.108492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.108463    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:06:49.134235 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:49.134200    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:49.230169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.230151    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:06:49.235078 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:49.235054    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Sep 15 09:06:49.244501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.244474    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:06:49.324622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.324564    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:49.341820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.341794    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 09:06:49.343162 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.343147    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Sep 15 09:06:49.350819 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.350801    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 09:06:49.775574 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:49.775540    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d0ab2acc9699061119b74368bceceb.slice/crio-e5742399422ecaf9b5a29950d4147753963ee19d1f7eb7bed83f183e2897c8b8 WatchSource:0}: Error finding container e5742399422ecaf9b5a29950d4147753963ee19d1f7eb7bed83f183e2897c8b8: Status 404 returned error can't find the container with id e5742399422ecaf9b5a29950d4147753963ee19d1f7eb7bed83f183e2897c8b8
Sep 15 09:06:49.901097 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.901059    2571 apiserver.go:52] "Watching apiserver"
Sep 15 09:06:49.912686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.912652    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 09:06:49.914041 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.914009    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-qb5dn","openshift-image-registry/node-ca-2494n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal","openshift-multus/multus-vh8mh","openshift-network-diagnostics/network-check-target-xd2s6","openshift-network-operator/iptables-alerter-fd6rx","kube-system/global-pull-secret-syncer-dqkpv","kube-system/konnectivity-agent-748rj","openshift-dns/node-resolver-gxw6x","openshift-multus/multus-additional-cni-plugins-bgrjz","openshift-multus/network-metrics-daemon-d8h8m","openshift-ovn-kubernetes/ovnkube-node-89jmt","kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"]
Sep 15 09:06:49.917050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.917026    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:49.917165 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:49.917121    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:06:49.919227 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.919206    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:49.919339 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:49.919284    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:06:49.921725 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.921700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.924135 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.924111    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 09:06:49.924410 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.924389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.925134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.925113    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-phrmm\""
Sep 15 09:06:49.925237 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.925142    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.925237 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.925198    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.925389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.925373    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 09:06:49.926724 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.926703    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:49.926823 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:49.926778    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:06:49.928811 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.928591    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 09:06:49.928811 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.928679    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dhpt6\""
Sep 15 09:06:49.928811 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.928678    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.929041 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.928858    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.929114 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.929098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:49.929617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.929599    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 09:06:49.931964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.931644    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:49.932168 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.932145    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 09:06:49.932341 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.932300    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x29pp\""
Sep 15 09:06:49.932443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.932431    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.932525 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.932506    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.934436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.934418    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:49.935067 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.935047    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.935292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.935129    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.935292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.935275    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lfc8t\""
Sep 15 09:06:49.936963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.936917    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:49.936963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.936955    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 09:06:49.937279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.937244    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 09:06:49.937869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.937850    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pw5mb\""
Sep 15 09:06:49.938097 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.937989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t746\" (UniqueName: \"kubernetes.io/projected/95e24fa5-d51c-4569-90bc-931616d9254b-kube-api-access-7t746\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.938097 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-kubelet\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938097 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938083    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-cni-bin\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938136    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-cni-multus\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-conf-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-etc-kubernetes\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-cnibin\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938261    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-cni-binary-copy\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-daemon-config\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.938349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.938314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-hostroot\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.939200 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939177    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrhm\" (UniqueName: \"kubernetes.io/projected/84527138-dafc-4435-b121-9215368bcb16-kube-api-access-hlrhm\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:49.939314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939289    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-sys-fs\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.939383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-cni-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.939445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-k8s-cni-cncf-io\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.939502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939478    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-netns\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939520    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22a36c08-dd46-4c1c-85a6-d53a59b56d41-dbus\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939721    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939766    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-socket-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939843    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-system-cni-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-socket-dir-parent\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.939930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-multus-certs\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-device-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22a36c08-dd46-4c1c-85a6-d53a59b56d41-kubelet-config\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-registration-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k2z\" (UniqueName: \"kubernetes.io/projected/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-kube-api-access-p4k2z\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940540    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.940971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.940964    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.941664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.941130    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 09:06:49.941664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.941337    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:49.941664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.941379    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vc2lf\""
Sep 15 09:06:49.944994 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.944973    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:49.945234 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.945214    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 09:06:49.945764 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.945710    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9bc2l\""
Sep 15 09:06:49.946039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.946020    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 09:06:49.948611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.948548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:49.948611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.948570    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.948611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.948596    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qpm6l\""
Sep 15 09:06:49.948804 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.948739    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.952128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.952111    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 09:06:49.952471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.952456    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 09:06:49.952774 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.952751    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 09:06:49.953119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.953100    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 09:06:49.953401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.953386    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 09:06:49.953615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.953600    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s8d48\""
Sep 15 09:06:49.956846 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.956827    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 09:06:49.961098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.961078    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 09:06:49.999957 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:49.999932    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fl6q6"
Sep 15 09:06:50.011845 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.011804    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fl6q6"
Sep 15 09:06:50.025124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.025047    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 09:06:50.041026 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.040990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-cni-multus\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041034    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-etc-kubernetes\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef628e49-f57f-43ab-a6df-2eed9e923881-hosts-file\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.041158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041102    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-daemon-config\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041121    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-etc-kubernetes\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-cni-multus\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvzp\" (UniqueName: \"kubernetes.io/projected/ef628e49-f57f-43ab-a6df-2eed9e923881-kube-api-access-bsvzp\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.041430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5428b578-20cb-414f-880f-63fe41a3a1da-etc-tuned\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.041430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-log-socket\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.041430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovnkube-config\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.041430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovn-node-metrics-cert\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.041430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrhm\" (UniqueName: \"kubernetes.io/projected/84527138-dafc-4435-b121-9215368bcb16-kube-api-access-hlrhm\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:50.041430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-cni-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-k8s-cni-cncf-io\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041492    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-iptables-alerter-script\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041519    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/931e8042-1c1c-4a89-9009-81d67154d70b-agent-certs\") pod \"konnectivity-agent-748rj\" (UID: \"931e8042-1c1c-4a89-9009-81d67154d70b\") " pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041545    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-systemd\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041567    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10e5b976-0d90-4ed2-a797-274776807382-serviceca\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041597    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-cnibin\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041626    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-cni-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.041710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041693    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-k8s-cni-cncf-io\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041726    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-netns\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041746    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-daemon-config\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041783    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-netns\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-socket-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041818    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-multus-certs\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.041877    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041938    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-run-multus-certs\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.041960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67599\" (UniqueName: \"kubernetes.io/projected/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-kube-api-access-67599\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.041984    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:50.541963097 +0000 UTC m=+3.001252639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef628e49-f57f-43ab-a6df-2eed9e923881-tmp-dir\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-socket-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-sys\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-kubelet\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.042172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042123    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-node-log\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042132    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-etc-selinux\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-cni-netd\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovnkube-script-lib\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042199    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22a36c08-dd46-4c1c-85a6-d53a59b56d41-kubelet-config\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-registration-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042263    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k2z\" (UniqueName: \"kubernetes.io/projected/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-kube-api-access-p4k2z\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042291    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/22a36c08-dd46-4c1c-85a6-d53a59b56d41-kubelet-config\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10e5b976-0d90-4ed2-a797-274776807382-host\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-modprobe-d\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysctl-d\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysctl-conf\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t746\" (UniqueName: \"kubernetes.io/projected/95e24fa5-d51c-4569-90bc-931616d9254b-kube-api-access-7t746\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-kubelet\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042493    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-registration-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-run\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.043079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-kubelet\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-run-netns\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-var-lib-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042810    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-kubernetes\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-cni-bin\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-conf-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042958    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-host-var-lib-cni-bin\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.042961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25cs\" (UniqueName: \"kubernetes.io/projected/32565e69-75c9-48aa-a026-ed63498f2fa8-kube-api-access-s25cs\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-conf-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043007    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-slash\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-etc-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043103    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-cnibin\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043128    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-cni-binary-copy\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043154    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-host\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-hostroot\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-sys-fs\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.043836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-cni-bin\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-sys-fs\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043213    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-cnibin\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043354    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-hostroot\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-host-slash\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrhd\" (UniqueName: \"kubernetes.io/projected/10e5b976-0d90-4ed2-a797-274776807382-kube-api-access-9xrhd\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043519    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/931e8042-1c1c-4a89-9009-81d67154d70b-konnectivity-ca\") pod \"konnectivity-agent-748rj\" (UID: \"931e8042-1c1c-4a89-9009-81d67154d70b\") " pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22a36c08-dd46-4c1c-85a6-d53a59b56d41-dbus\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-system-cni-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-socket-dir-parent\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043693    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/22a36c08-dd46-4c1c-85a6-d53a59b56d41-dbus\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043687    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-multus-socket-dir-parent\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043700    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-system-cni-dir\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-system-cni-dir\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043749    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.044582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043730    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-cni-binary-copy\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043765    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-systemd-units\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-systemd\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-device-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043899    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjs6\" (UniqueName: \"kubernetes.io/projected/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-kube-api-access-pjjs6\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.043926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysconfig\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044030    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95e24fa5-d51c-4569-90bc-931616d9254b-device-dir\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044038    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-lib-modules\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044062    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5428b578-20cb-414f-880f-63fe41a3a1da-tmp\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2p95\" (UniqueName: \"kubernetes.io/projected/5428b578-20cb-414f-880f-63fe41a3a1da-kube-api-access-t2p95\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-env-overrides\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-var-lib-kubelet\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044218    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-ovn\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.044242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-run-ovn-kubernetes\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.045428 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.044326    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:50.046272 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.044565    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:50.544544785 +0000 UTC m=+3.003834323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:50.049696 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.049335    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 09:06:50.054542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.053787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t746\" (UniqueName: \"kubernetes.io/projected/95e24fa5-d51c-4569-90bc-931616d9254b-kube-api-access-7t746\") pod \"aws-ebs-csi-driver-node-gdnw2\" (UID: \"95e24fa5-d51c-4569-90bc-931616d9254b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.054542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.053970    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrhm\" (UniqueName: \"kubernetes.io/projected/84527138-dafc-4435-b121-9215368bcb16-kube-api-access-hlrhm\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:50.054542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.054013    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 09:06:50.054542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.054484    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k2z\" (UniqueName: \"kubernetes.io/projected/ad9d08e1-762d-46c7-bf7c-0723d12e33cb-kube-api-access-p4k2z\") pod \"multus-vh8mh\" (UID: \"ad9d08e1-762d-46c7-bf7c-0723d12e33cb\") " pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.055398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.055337    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" event={"ID":"c3d0ab2acc9699061119b74368bceceb","Type":"ContainerStarted","Data":"e5742399422ecaf9b5a29950d4147753963ee19d1f7eb7bed83f183e2897c8b8"}
Sep 15 09:06:50.056800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.056764    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal" event={"ID":"ff78554c81b115a40bbe63702b1bfba3","Type":"ContainerStarted","Data":"57c824ad28f737ae778efad83a0408ffff3ac651cf1a69f536617e709af6f097"}
Sep 15 09:06:50.145185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjs6\" (UniqueName: \"kubernetes.io/projected/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-kube-api-access-pjjs6\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.145185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145199    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysconfig\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-lib-modules\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145283    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5428b578-20cb-414f-880f-63fe41a3a1da-tmp\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145308    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2p95\" (UniqueName: \"kubernetes.io/projected/5428b578-20cb-414f-880f-63fe41a3a1da-kube-api-access-t2p95\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145333    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-env-overrides\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-var-lib-kubelet\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145360    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysconfig\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145398    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.145449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145423    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-ovn\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-run-ovn-kubernetes\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef628e49-f57f-43ab-a6df-2eed9e923881-hosts-file\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvzp\" (UniqueName: \"kubernetes.io/projected/ef628e49-f57f-43ab-a6df-2eed9e923881-kube-api-access-bsvzp\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145542    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5428b578-20cb-414f-880f-63fe41a3a1da-etc-tuned\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-log-socket\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovnkube-config\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovn-node-metrics-cert\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-iptables-alerter-script\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.145966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.145756    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef628e49-f57f-43ab-a6df-2eed9e923881-hosts-file\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146037    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-env-overrides\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146099    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-log-socket\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-ovn\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146324    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-lib-modules\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146372    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-run-ovn-kubernetes\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.146502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146387    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-iptables-alerter-script\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146606    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146655    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovnkube-config\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-var-lib-kubelet\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/931e8042-1c1c-4a89-9009-81d67154d70b-agent-certs\") pod \"konnectivity-agent-748rj\" (UID: \"931e8042-1c1c-4a89-9009-81d67154d70b\") " pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146760    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-systemd\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10e5b976-0d90-4ed2-a797-274776807382-serviceca\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-cnibin\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.146840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.147288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67599\" (UniqueName: \"kubernetes.io/projected/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-kube-api-access-67599\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.147288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.146890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef628e49-f57f-43ab-a6df-2eed9e923881-tmp-dir\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.147288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147132    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef628e49-f57f-43ab-a6df-2eed9e923881-tmp-dir\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.147288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147222    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-cnibin\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.147500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10e5b976-0d90-4ed2-a797-274776807382-serviceca\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.147500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147373    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-sys\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.147500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147402    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-kubelet\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.147500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.147500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-node-log\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.147500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-cni-netd\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovnkube-script-lib\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10e5b976-0d90-4ed2-a797-274776807382-host\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147592    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-modprobe-d\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysctl-d\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysctl-conf\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32565e69-75c9-48aa-a026-ed63498f2fa8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147731    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-systemd\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147736    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-run\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.147800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147781    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-sys\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147820    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-kubelet\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147943    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-node-log\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-run\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147990    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-cni-netd\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.147991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-run-netns\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-var-lib-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148079    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-kubernetes\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.148385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25cs\" (UniqueName: \"kubernetes.io/projected/32565e69-75c9-48aa-a026-ed63498f2fa8-kube-api-access-s25cs\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.148872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovnkube-script-lib\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.148872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148603    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5428b578-20cb-414f-880f-63fe41a3a1da-etc-tuned\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.148872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10e5b976-0d90-4ed2-a797-274776807382-host\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.148872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148732    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.148872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysctl-d\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.149161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-sysctl-conf\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.149161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-modprobe-d\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.149161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-etc-kubernetes\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.149161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148972    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.149161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.149029    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-run-netns\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.149392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.149251    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-var-lib-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.149392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.149306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-slash\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.150091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.150054    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5428b578-20cb-414f-880f-63fe41a3a1da-tmp\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.150387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.150365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-ovn-node-metrics-cert\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.148131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-slash\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-etc-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151333    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-host\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.151377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-cni-bin\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151390    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-host-slash\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrhd\" (UniqueName: \"kubernetes.io/projected/10e5b976-0d90-4ed2-a797-274776807382-kube-api-access-9xrhd\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/931e8042-1c1c-4a89-9009-81d67154d70b-konnectivity-ca\") pod \"konnectivity-agent-748rj\" (UID: \"931e8042-1c1c-4a89-9009-81d67154d70b\") " pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-system-cni-dir\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151498    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-host-cni-bin\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-systemd-units\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-systemd\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.151557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151554    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-etc-openvswitch\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.152046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151817    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-host-slash\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.152046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151818    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-run-systemd\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.152046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32565e69-75c9-48aa-a026-ed63498f2fa8-system-cni-dir\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.152046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151906    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5428b578-20cb-414f-880f-63fe41a3a1da-host\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.152046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.151946    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-systemd-units\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.152290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.152171    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/931e8042-1c1c-4a89-9009-81d67154d70b-konnectivity-ca\") pod \"konnectivity-agent-748rj\" (UID: \"931e8042-1c1c-4a89-9009-81d67154d70b\") " pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.155677 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.155241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/931e8042-1c1c-4a89-9009-81d67154d70b-agent-certs\") pod \"konnectivity-agent-748rj\" (UID: \"931e8042-1c1c-4a89-9009-81d67154d70b\") " pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.155782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.155689    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjs6\" (UniqueName: \"kubernetes.io/projected/d3380af8-e22f-4ebf-9001-1e92a1e9ea8b-kube-api-access-pjjs6\") pod \"iptables-alerter-fd6rx\" (UID: \"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b\") " pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.156296 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.156269    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67599\" (UniqueName: \"kubernetes.io/projected/b5a20ed4-a5e3-45e6-ac11-080f61eb4968-kube-api-access-67599\") pod \"ovnkube-node-89jmt\" (UID: \"b5a20ed4-a5e3-45e6-ac11-080f61eb4968\") " pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.156296 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.156268    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvzp\" (UniqueName: \"kubernetes.io/projected/ef628e49-f57f-43ab-a6df-2eed9e923881-kube-api-access-bsvzp\") pod \"node-resolver-gxw6x\" (UID: \"ef628e49-f57f-43ab-a6df-2eed9e923881\") " pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.158338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.158316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2p95\" (UniqueName: \"kubernetes.io/projected/5428b578-20cb-414f-880f-63fe41a3a1da-kube-api-access-t2p95\") pod \"tuned-qb5dn\" (UID: \"5428b578-20cb-414f-880f-63fe41a3a1da\") " pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.158654 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.158491    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:06:50.158654 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.158515    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:06:50.158654 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.158530    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:50.158654 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.158608    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:50.658589617 +0000 UTC m=+3.117879156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:50.160868 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.160839    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25cs\" (UniqueName: \"kubernetes.io/projected/32565e69-75c9-48aa-a026-ed63498f2fa8-kube-api-access-s25cs\") pod \"multus-additional-cni-plugins-bgrjz\" (UID: \"32565e69-75c9-48aa-a026-ed63498f2fa8\") " pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.161357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.161338    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrhd\" (UniqueName: \"kubernetes.io/projected/10e5b976-0d90-4ed2-a797-274776807382-kube-api-access-9xrhd\") pod \"node-ca-2494n\" (UID: \"10e5b976-0d90-4ed2-a797-274776807382\") " pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.234167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.234124    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2"
Sep 15 09:06:50.243471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.243433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vh8mh"
Sep 15 09:06:50.252846 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.252806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fd6rx"
Sep 15 09:06:50.259599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.259576    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxw6x"
Sep 15 09:06:50.267463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.267430    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:06:50.275459 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.275349    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2494n"
Sep 15 09:06:50.283357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.283333    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgrjz"
Sep 15 09:06:50.289357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.289331    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qb5dn"
Sep 15 09:06:50.295193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.295173    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:06:50.555496 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.555408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:50.555496 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.555466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:50.555738 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.555580    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:50.555738 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.555647    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:51.555633247 +0000 UTC m=+4.014922772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:50.555738 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.555580    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:50.555738 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.555714    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:51.555701547 +0000 UTC m=+4.014991067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:50.756958 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:50.756904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:50.757137 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.757114    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:06:50.757137 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.757137    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:06:50.757258 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.757151    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:50.757258 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:50.757207    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:51.757193657 +0000 UTC m=+4.216483183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:50.922628 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.922580    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931e8042_1c1c_4a89_9009_81d67154d70b.slice/crio-3ca103cc88bba7b7b9d810e372946b9fc628aba65d0ae15c8c32f372a3ca69b4 WatchSource:0}: Error finding container 3ca103cc88bba7b7b9d810e372946b9fc628aba65d0ae15c8c32f372a3ca69b4: Status 404 returned error can't find the container with id 3ca103cc88bba7b7b9d810e372946b9fc628aba65d0ae15c8c32f372a3ca69b4
Sep 15 09:06:50.925502 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.925448    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a20ed4_a5e3_45e6_ac11_080f61eb4968.slice/crio-602de000f0f495e3df0c5bd48424d6ca5b064e807ce84b01bef0c6942ed6df4f WatchSource:0}: Error finding container 602de000f0f495e3df0c5bd48424d6ca5b064e807ce84b01bef0c6942ed6df4f: Status 404 returned error can't find the container with id 602de000f0f495e3df0c5bd48424d6ca5b064e807ce84b01bef0c6942ed6df4f
Sep 15 09:06:50.929008 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.928970    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e5b976_0d90_4ed2_a797_274776807382.slice/crio-160013e718430d2528962d5bbb22ccafa1d877527ea644b31a8fea1941241db6 WatchSource:0}: Error finding container 160013e718430d2528962d5bbb22ccafa1d877527ea644b31a8fea1941241db6: Status 404 returned error can't find the container with id 160013e718430d2528962d5bbb22ccafa1d877527ea644b31a8fea1941241db6
Sep 15 09:06:50.930444 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.930413    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3380af8_e22f_4ebf_9001_1e92a1e9ea8b.slice/crio-e0994a4649d0ef54bae041d0527230e3978bbbd6f6391f6fc44a70bb1922b263 WatchSource:0}: Error finding container e0994a4649d0ef54bae041d0527230e3978bbbd6f6391f6fc44a70bb1922b263: Status 404 returned error can't find the container with id e0994a4649d0ef54bae041d0527230e3978bbbd6f6391f6fc44a70bb1922b263
Sep 15 09:06:50.930838 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.930814    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef628e49_f57f_43ab_a6df_2eed9e923881.slice/crio-2c55e3c54940d82beb26499dffe00a01e3cea7bec1cd66d8b341f7c1fb6d8e3b WatchSource:0}: Error finding container 2c55e3c54940d82beb26499dffe00a01e3cea7bec1cd66d8b341f7c1fb6d8e3b: Status 404 returned error can't find the container with id 2c55e3c54940d82beb26499dffe00a01e3cea7bec1cd66d8b341f7c1fb6d8e3b
Sep 15 09:06:50.953929 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.953893    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9d08e1_762d_46c7_bf7c_0723d12e33cb.slice/crio-484de9f130a63ab5db89b52b295a0c6af16654761f13951f7088005093b10d72 WatchSource:0}: Error finding container 484de9f130a63ab5db89b52b295a0c6af16654761f13951f7088005093b10d72: Status 404 returned error can't find the container with id 484de9f130a63ab5db89b52b295a0c6af16654761f13951f7088005093b10d72
Sep 15 09:06:50.955303 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.955267    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e24fa5_d51c_4569_90bc_931616d9254b.slice/crio-e897bb6076637fa4afa1028ec05c37e34b2234ea636254b95b7a23a976891922 WatchSource:0}: Error finding container e897bb6076637fa4afa1028ec05c37e34b2234ea636254b95b7a23a976891922: Status 404 returned error can't find the container with id e897bb6076637fa4afa1028ec05c37e34b2234ea636254b95b7a23a976891922
Sep 15 09:06:50.956163 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.956101    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32565e69_75c9_48aa_a026_ed63498f2fa8.slice/crio-1bdc6d8b18b59b4437c54921abd901877bcd2b3d70306b6bd88e4cb8c73e4161 WatchSource:0}: Error finding container 1bdc6d8b18b59b4437c54921abd901877bcd2b3d70306b6bd88e4cb8c73e4161: Status 404 returned error can't find the container with id 1bdc6d8b18b59b4437c54921abd901877bcd2b3d70306b6bd88e4cb8c73e4161
Sep 15 09:06:50.957231 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:06:50.957201    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5428b578_20cb_414f_880f_63fe41a3a1da.slice/crio-676044d8cb92eff93a6818990f244a4580649ae4ab629d93223268c57d3cf977 WatchSource:0}: Error finding container 676044d8cb92eff93a6818990f244a4580649ae4ab629d93223268c57d3cf977: Status 404 returned error can't find the container with id 676044d8cb92eff93a6818990f244a4580649ae4ab629d93223268c57d3cf977
Sep 15 09:06:51.015464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.015227    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 09:01:50 +0000 UTC" deadline="2028-05-20 06:34:37.830554607 +0000 UTC"
Sep 15 09:06:51.015464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.015273    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14709h27m46.815283794s"
Sep 15 09:06:51.060871 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.060817    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2494n" event={"ID":"10e5b976-0d90-4ed2-a797-274776807382","Type":"ContainerStarted","Data":"160013e718430d2528962d5bbb22ccafa1d877527ea644b31a8fea1941241db6"}
Sep 15 09:06:51.062441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.062397    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxw6x" event={"ID":"ef628e49-f57f-43ab-a6df-2eed9e923881","Type":"ContainerStarted","Data":"2c55e3c54940d82beb26499dffe00a01e3cea7bec1cd66d8b341f7c1fb6d8e3b"}
Sep 15 09:06:51.063799 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.063774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fd6rx" event={"ID":"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b","Type":"ContainerStarted","Data":"e0994a4649d0ef54bae041d0527230e3978bbbd6f6391f6fc44a70bb1922b263"}
Sep 15 09:06:51.065222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.065197    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-748rj" event={"ID":"931e8042-1c1c-4a89-9009-81d67154d70b","Type":"ContainerStarted","Data":"3ca103cc88bba7b7b9d810e372946b9fc628aba65d0ae15c8c32f372a3ca69b4"}
Sep 15 09:06:51.067437 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.067408    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qb5dn" event={"ID":"5428b578-20cb-414f-880f-63fe41a3a1da","Type":"ContainerStarted","Data":"676044d8cb92eff93a6818990f244a4580649ae4ab629d93223268c57d3cf977"}
Sep 15 09:06:51.068866 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.068843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerStarted","Data":"1bdc6d8b18b59b4437c54921abd901877bcd2b3d70306b6bd88e4cb8c73e4161"}
Sep 15 09:06:51.070262 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.070230    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"602de000f0f495e3df0c5bd48424d6ca5b064e807ce84b01bef0c6942ed6df4f"}
Sep 15 09:06:51.074113 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.074082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2" event={"ID":"95e24fa5-d51c-4569-90bc-931616d9254b","Type":"ContainerStarted","Data":"e897bb6076637fa4afa1028ec05c37e34b2234ea636254b95b7a23a976891922"}
Sep 15 09:06:51.075419 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.075392    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vh8mh" event={"ID":"ad9d08e1-762d-46c7-bf7c-0723d12e33cb","Type":"ContainerStarted","Data":"484de9f130a63ab5db89b52b295a0c6af16654761f13951f7088005093b10d72"}
Sep 15 09:06:51.563556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.563520    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:51.563738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.563616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:51.563738 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.563710    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:51.563738 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.563711    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:51.563876 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.563774    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:53.563754466 +0000 UTC m=+6.023043987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:51.563876 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.563790    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:53.563783314 +0000 UTC m=+6.023072835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:51.766119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:51.765384    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:51.766119 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.765582    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:06:51.766119 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.765616    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:06:51.766119 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.765631    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:51.766119 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:51.765698    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:53.765677762 +0000 UTC m=+6.224967304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:52.018099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.017971    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 09:01:50 +0000 UTC" deadline="2028-06-25 08:51:24.646807634 +0000 UTC"
Sep 15 09:06:52.018099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.018015    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15575h44m32.628796925s"
Sep 15 09:06:52.050765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.049999    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:52.050765 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:52.050155    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:06:52.050765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.050611    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:52.050765 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:52.050707    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:06:52.051441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.051290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:52.051441 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:52.051387    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:06:52.101493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.101450    2571 generic.go:356] "Generic (PLEG): container finished" podID="c3d0ab2acc9699061119b74368bceceb" containerID="60a981b7204f92c21ba860c94a278606a1bc66b3941c5b708ca4932bcb54375e" exitCode=0
Sep 15 09:06:52.101694 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.101573    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" event={"ID":"c3d0ab2acc9699061119b74368bceceb","Type":"ContainerDied","Data":"60a981b7204f92c21ba860c94a278606a1bc66b3941c5b708ca4932bcb54375e"}
Sep 15 09:06:52.124403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.122967    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal" podStartSLOduration=2.122943701 podStartE2EDuration="2.122943701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:06:51.089204609 +0000 UTC m=+3.548494154" watchObservedRunningTime="2026-09-15 09:06:52.122943701 +0000 UTC m=+4.582233246"
Sep 15 09:06:52.124403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:52.123862    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal" event={"ID":"ff78554c81b115a40bbe63702b1bfba3","Type":"ContainerStarted","Data":"9ef9a491c890794db859683030d6600c2be7fc7e327db906fecc405db164d1af"}
Sep 15 09:06:53.159468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:53.159425    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" event={"ID":"c3d0ab2acc9699061119b74368bceceb","Type":"ContainerStarted","Data":"16765cd8a91fc5ca7f13db6a970b14167b4374698dfc4a5114e48dde83de585e"}
Sep 15 09:06:53.187880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:53.187805    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" podStartSLOduration=3.18778144 podStartE2EDuration="3.18778144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:06:53.182948524 +0000 UTC m=+5.642238068" watchObservedRunningTime="2026-09-15 09:06:53.18778144 +0000 UTC m=+5.647070986"
Sep 15 09:06:53.580148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:53.579989    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:53.580148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:53.580072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:53.580441 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.580272    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:53.580441 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.580344    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:57.580322248 +0000 UTC m=+10.039611785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:53.580934 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.580891    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:53.581022 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.580977    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:57.580958433 +0000 UTC m=+10.040247955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:53.781986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:53.781891    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:53.782204 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.782065    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:06:53.782204 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.782091    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:06:53.782204 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.782114    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:53.782204 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:53.782176    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:06:57.782157678 +0000 UTC m=+10.241447218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:54.050119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:54.050003    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:54.050119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:54.050060    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:54.050386 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:54.050158    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:06:54.050386 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:54.050248    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:06:54.050386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:54.050315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:54.050548 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:54.050385    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:06:56.052557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:56.051665    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:56.052557 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:56.051802    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:06:56.052557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:56.052217    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:56.052557 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:56.052361    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:06:56.052557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:56.052416    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:56.052557 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:56.052498    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:06:57.619382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:57.619107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:57.619382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:57.619214    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:57.619382 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.619228    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:57.619382 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.619305    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:05.619281078 +0000 UTC m=+18.078570620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:06:57.619382 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.619316    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:57.619382 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.619351    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:05.619341102 +0000 UTC m=+18.078630638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:06:57.820741 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:57.820617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:57.820984 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.820844    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:06:57.820984 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.820889    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:06:57.820984 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.820904    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:57.820984 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:57.820968    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:05.820948705 +0000 UTC m=+18.280238241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:06:58.050505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:58.050395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:06:58.050668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:58.050535    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:06:58.050668 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:58.050544    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:06:58.050798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:06:58.050678    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:06:58.050798 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:58.050691    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:06:58.050798 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:06:58.050770    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:00.049468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:00.049367    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:00.050005 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:00.049366    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:00.050005 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:00.049512    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:00.050005 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:00.049525    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:00.050005 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:00.049595    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:00.050005 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:00.049696    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:02.049783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:02.049721    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:02.050157 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:02.049877    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:02.050266 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:02.049721    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:02.050390 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:02.050369    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:02.050728 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:02.050693    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:02.050864 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:02.050794    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:04.049134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:04.049098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:04.049582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:04.049099    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:04.049582 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:04.049241    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:04.049582 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:04.049302    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:04.049582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:04.049105    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:04.049582 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:04.049376    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:05.677424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:05.677363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:05.677875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:05.677435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:05.677875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.677534    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:07:05.677875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.677545    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:07:05.677875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.677598    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:21.677583252 +0000 UTC m=+34.136872771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:07:05.677875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.677612    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:21.677606059 +0000 UTC m=+34.136895582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:07:05.879340 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:05.879296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:05.879546 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.879455    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:07:05.879546 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.879482    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:07:05.879546 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.879494    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:07:05.879701 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:05.879559    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:21.879541287 +0000 UTC m=+34.338830823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:07:06.049618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:06.049541    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:06.049779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:06.049541    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:06.049779 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:06.049658    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:06.049779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:06.049677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:06.049779 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:06.049765    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:06.050006 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:06.049883    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:08.049672 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:08.049626    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:08.050192 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:08.049781    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:08.050192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:08.049873    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:08.050192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:08.049975    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:08.050192 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:08.049991    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:08.050192 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:08.050085    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:10.049398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.049366    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:10.049969 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:10.049506    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:10.049969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.049583    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:10.049969 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:10.049655    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:10.049969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.049781    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:10.049969 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:10.049874    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:10.206428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.206393    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"5b7304721a6cfdfdcab50a44723e7458538c14678e7775078735bb0538a6a8e4"}
Sep 15 09:07:10.206596 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.206529    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"34d31c13c384d3c07b25b9f842d4726726b86ad82bf424d24e6871205cc19c3e"}
Sep 15 09:07:10.206596 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.206586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"dd467eafa561aa41fdb9067bd2b8d670cda367c25062273a96e65dd8f4187f1b"}
Sep 15 09:07:10.208221 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.208190    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2" event={"ID":"95e24fa5-d51c-4569-90bc-931616d9254b","Type":"ContainerStarted","Data":"ed30f692aeaf9dd1b8ff6a3274c4eb8c63cb889f27213fc809a9f505d53dfa1a"}
Sep 15 09:07:10.209893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.209866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vh8mh" event={"ID":"ad9d08e1-762d-46c7-bf7c-0723d12e33cb","Type":"ContainerStarted","Data":"337ccb155b770d1f75767a455071f1c4e80827cb01d601e6593a7421d6fd3258"}
Sep 15 09:07:10.211428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.211403    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2494n" event={"ID":"10e5b976-0d90-4ed2-a797-274776807382","Type":"ContainerStarted","Data":"346605a25699781ea0146fbfa815cb51c41537572954281c07af51f1fda2c864"}
Sep 15 09:07:10.212969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.212935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxw6x" event={"ID":"ef628e49-f57f-43ab-a6df-2eed9e923881","Type":"ContainerStarted","Data":"56a85d61b69c636b3d23f05ea758821d770515c53e72e22c0eae76eb16628d87"}
Sep 15 09:07:10.214408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.214384    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-748rj" event={"ID":"931e8042-1c1c-4a89-9009-81d67154d70b","Type":"ContainerStarted","Data":"c256d8069129c467e80e2718f28d6194fd4ec24fea99b33fed1dd546c33a66a0"}
Sep 15 09:07:10.215914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.215894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qb5dn" event={"ID":"5428b578-20cb-414f-880f-63fe41a3a1da","Type":"ContainerStarted","Data":"02830df74153a1e789b13ff180d561db82b73baf08d0aac29f0ae668724729e2"}
Sep 15 09:07:10.217579 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.217553    2571 generic.go:356] "Generic (PLEG): container finished" podID="32565e69-75c9-48aa-a026-ed63498f2fa8" containerID="09e15abe8db22413261eaafc3aa3193a8cb3e2041b66328ef267796c3294db3c" exitCode=0
Sep 15 09:07:10.217660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.217594    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerDied","Data":"09e15abe8db22413261eaafc3aa3193a8cb3e2041b66328ef267796c3294db3c"}
Sep 15 09:07:10.226523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.226477    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vh8mh" podStartSLOduration=2.2986113440000002 podStartE2EDuration="21.226463598s" totalImagesPullingTime="18.927852254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:10.22632291 +0000 UTC m=+22.685612451" watchObservedRunningTime="2026-09-15 09:07:10.226463598 +0000 UTC m=+22.685753136"
Sep 15 09:07:10.241384 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.241337    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gxw6x" podStartSLOduration=2.362961839 podStartE2EDuration="21.241320743s" totalImagesPullingTime="18.878358904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:10.24123637 +0000 UTC m=+22.700525915" watchObservedRunningTime="2026-09-15 09:07:10.241320743 +0000 UTC m=+22.700610295"
Sep 15 09:07:10.255221 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.255168    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-748rj" podStartSLOduration=2.3493246660000002 podStartE2EDuration="21.255151399s" totalImagesPullingTime="18.905826733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:10.255098368 +0000 UTC m=+22.714387909" watchObservedRunningTime="2026-09-15 09:07:10.255151399 +0000 UTC m=+22.714440940"
Sep 15 09:07:10.269146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.269100    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2494n" podStartSLOduration=9.752802024 podStartE2EDuration="21.269083828s" totalImagesPullingTime="11.516281804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:10.268926202 +0000 UTC m=+22.728215746" watchObservedRunningTime="2026-09-15 09:07:10.269083828 +0000 UTC m=+22.728373368"
Sep 15 09:07:10.305381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:10.305325    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qb5dn" podStartSLOduration=2.4385137 podStartE2EDuration="21.305308851s" totalImagesPullingTime="18.866795151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:10.305277775 +0000 UTC m=+22.764567315" watchObservedRunningTime="2026-09-15 09:07:10.305308851 +0000 UTC m=+22.764598392"
Sep 15 09:07:11.222011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:11.221971    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fd6rx" event={"ID":"d3380af8-e22f-4ebf-9001-1e92a1e9ea8b","Type":"ContainerStarted","Data":"bb71f7e0bb220a5ceb922dacd16c5e5b1ddf808e6503adcbeb7a08fc24f808f8"}
Sep 15 09:07:11.226370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:11.226338    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"c659c7a2e0834db5016a310b2df80770a95e08ce3f88685660728cab70d1d549"}
Sep 15 09:07:11.226499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:11.226414    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"abb2f6bb6715ee59d309f4456d5bf1411f92983fe09b2367905b80a44f273974"}
Sep 15 09:07:11.227212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:11.227163    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"b86b2addea0a3bf8ce74fb9b31f57903f9d82f4e1a2a4de2d530787e204e2bc6"}
Sep 15 09:07:11.237119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:11.237044    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fd6rx" podStartSLOduration=3.358242846 podStartE2EDuration="22.237023294s" totalImagesPullingTime="18.878780448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:11.236643924 +0000 UTC m=+23.695933467" watchObservedRunningTime="2026-09-15 09:07:11.237023294 +0000 UTC m=+23.696312836"
Sep 15 09:07:11.866700 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:11.866670    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 09:07:12.009943 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.009762    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T09:07:11.866692557Z","UUID":"4a94c995-85fe-47f7-a271-386914fb8ada","Handler":null,"Name":"","Endpoint":""}
Sep 15 09:07:12.011482 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.011456    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 09:07:12.011615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.011494    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 09:07:12.049541 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.049505    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:12.049714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.049552    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:12.049714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.049505    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:12.049714 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:12.049656    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:12.049888 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:12.049752    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:12.049888 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:12.049856    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:12.230026 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:12.229989    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2" event={"ID":"95e24fa5-d51c-4569-90bc-931616d9254b","Type":"ContainerStarted","Data":"cbfabf571b3593671ee318dbcba21dec5d7663bcef94f11fa20b676d87c41d9a"}
Sep 15 09:07:13.235514 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:13.235483    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"29019615c4bf60b60736bd01bcf3cb4decd219605a36c3f6db2ac628ee16ee01"}
Sep 15 09:07:14.049192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:14.049152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:14.049365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:14.049152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:14.049365 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:14.049298    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:14.049365 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:14.049333    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:14.049365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:14.049152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:14.049582 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:14.049405    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:15.192952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.192747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:07:15.193653 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.193313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:07:15.242144 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.242063    2571 generic.go:356] "Generic (PLEG): container finished" podID="32565e69-75c9-48aa-a026-ed63498f2fa8" containerID="109c0d54717c4807e6fd35d8a0a9aa58e8cf00268dd1993724c6231761e163ae" exitCode=0
Sep 15 09:07:15.242289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.242147    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerDied","Data":"109c0d54717c4807e6fd35d8a0a9aa58e8cf00268dd1993724c6231761e163ae"}
Sep 15 09:07:15.246091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.246063    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" event={"ID":"b5a20ed4-a5e3-45e6-ac11-080f61eb4968","Type":"ContainerStarted","Data":"af13496260554aa7c8b0535f4645265ecabed11ed49f9eea9bf7d0c54c0b438a"}
Sep 15 09:07:15.246392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.246358    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:07:15.246472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.246433    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:07:15.246523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.246484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:07:15.249299 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.249263    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2" event={"ID":"95e24fa5-d51c-4569-90bc-931616d9254b","Type":"ContainerStarted","Data":"3aadf1be0bf17b8c34c0b2ef45601b2066729c23512ae5675eb0a47a824e170a"}
Sep 15 09:07:15.249634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.249516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:07:15.249921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.249904    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-748rj"
Sep 15 09:07:15.262016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.261976    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:07:15.262140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.262082    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:07:15.288459 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.288415    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gdnw2" podStartSLOduration=2.756104885 podStartE2EDuration="26.288399133s" totalImagesPullingTime="23.532294248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:15.288289473 +0000 UTC m=+27.747579015" watchObservedRunningTime="2026-09-15 09:07:15.288399133 +0000 UTC m=+27.747688675"
Sep 15 09:07:15.322421 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:15.322366    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt" podStartSLOduration=7.360387822 podStartE2EDuration="26.322352428s" totalImagesPullingTime="18.961964606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:15.32181807 +0000 UTC m=+27.781107612" watchObservedRunningTime="2026-09-15 09:07:15.322352428 +0000 UTC m=+27.781641968"
Sep 15 09:07:16.049484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.049453    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:16.049645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.049450    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:16.049645 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:16.049607    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:16.049732 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.049450    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:16.049732 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:16.049649    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:16.049805 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:16.049752    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:16.254188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.254145    2571 generic.go:356] "Generic (PLEG): container finished" podID="32565e69-75c9-48aa-a026-ed63498f2fa8" containerID="a057e5cbbaf58f87305a198e8b934455914a534a4df961f1acde0361f076ba6d" exitCode=0
Sep 15 09:07:16.255096 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.255068    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerDied","Data":"a057e5cbbaf58f87305a198e8b934455914a534a4df961f1acde0361f076ba6d"}
Sep 15 09:07:16.905885 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.905627    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xd2s6"]
Sep 15 09:07:16.905885 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.905797    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:16.906100 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:16.905925    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:16.908998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.908967    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dqkpv"]
Sep 15 09:07:16.909190 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.909097    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:16.909254 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:16.909207    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:16.909722 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.909697    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8h8m"]
Sep 15 09:07:16.909854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:16.909800    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:16.909965 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:16.909933    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:17.259216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:17.259131    2571 generic.go:356] "Generic (PLEG): container finished" podID="32565e69-75c9-48aa-a026-ed63498f2fa8" containerID="1c23b289127b6f4174e50ddba902a6eed3bfe8a85b09902aa2afcdd7788f2c08" exitCode=0
Sep 15 09:07:17.259552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:17.259221    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerDied","Data":"1c23b289127b6f4174e50ddba902a6eed3bfe8a85b09902aa2afcdd7788f2c08"}
Sep 15 09:07:18.050231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:18.050194    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:18.050423 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:18.050293    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:19.049244 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:19.049213    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:19.049244 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:19.049254    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:19.049761 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:19.049355    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:19.049761 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:19.049503    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:20.048840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:20.048806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:20.049027 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:20.048947    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:21.048968 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:21.048933    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:21.049463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:21.048933    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:21.049463 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.049045    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqkpv" podUID="22a36c08-dd46-4c1c-85a6-d53a59b56d41"
Sep 15 09:07:21.049463 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.049104    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8h8m" podUID="84527138-dafc-4435-b121-9215368bcb16"
Sep 15 09:07:21.697515 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:21.697472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:21.697690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:21.697533    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:21.697690 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.697632    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:07:21.697790 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.697715    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:53.697694362 +0000 UTC m=+66.156983904 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 09:07:21.697790 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.697642    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 09:07:21.697899 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.697791    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret podName:22a36c08-dd46-4c1c-85a6-d53a59b56d41 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:53.69777527 +0000 UTC m=+66.157064795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret") pod "global-pull-secret-syncer-dqkpv" (UID: "22a36c08-dd46-4c1c-85a6-d53a59b56d41") : object "kube-system"/"original-pull-secret" not registered
Sep 15 09:07:21.898678 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:21.898632    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:21.898895 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.898850    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 09:07:21.898895 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.898874    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 09:07:21.898895 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.898886    2571 projected.go:196] Error preparing data for projected volume kube-api-access-fsljb for pod openshift-network-diagnostics/network-check-target-xd2s6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:07:21.899055 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:21.898955    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb podName:7580fb52-d812-428d-b561-f5cd4e97fcf4 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:53.898934789 +0000 UTC m=+66.358224327 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fsljb" (UniqueName: "kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb") pod "network-check-target-xd2s6" (UID: "7580fb52-d812-428d-b561-f5cd4e97fcf4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 09:07:22.049194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.049159    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:22.049564 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:22.049292    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xd2s6" podUID="7580fb52-d812-428d-b561-f5cd4e97fcf4"
Sep 15 09:07:22.774766 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.774595    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeReady"
Sep 15 09:07:22.774944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.774874    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 09:07:22.835640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.835609    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-785dcc4fcf-b6mzw"]
Sep 15 09:07:22.850138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.850110    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vc4p7"]
Sep 15 09:07:22.850304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.850242    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:22.852982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.852957    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 09:07:22.852982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.852959    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 09:07:22.853178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.853034    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 09:07:22.853415 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.853395    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8vjg9\""
Sep 15 09:07:22.858696 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.858674    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 09:07:22.863533 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.863502    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qwp25"]
Sep 15 09:07:22.863669 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.863641    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:22.866194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.866092    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 09:07:22.866194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.866119    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 09:07:22.866367 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.866095    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lb4vr\""
Sep 15 09:07:22.883523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.883489    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-785dcc4fcf-b6mzw"]
Sep 15 09:07:22.883523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.883530    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vc4p7"]
Sep 15 09:07:22.883737 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.883557    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qwp25"]
Sep 15 09:07:22.883737 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.883600    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:22.886417 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.886394    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 09:07:22.886547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.886455    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 09:07:22.886547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.886471    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2z5lh\""
Sep 15 09:07:22.886640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.886556    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 09:07:22.994006 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:22.993922    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:23.007403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007378    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdc7ebe9-46d4-4905-af95-2ae799a30157-cert\") pod \"ingress-canary-qwp25\" (UID: \"fdc7ebe9-46d4-4905-af95-2ae799a30157\") " pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.007517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007408    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cz9w\" (UniqueName: \"kubernetes.io/projected/fdc7ebe9-46d4-4905-af95-2ae799a30157-kube-api-access-5cz9w\") pod \"ingress-canary-qwp25\" (UID: \"fdc7ebe9-46d4-4905-af95-2ae799a30157\") " pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.007517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.007517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-image-registry-private-configuration\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd2aced5-6d03-4660-9c20-cf2248046c9c-ca-trust-extracted\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007510    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-bound-sa-token\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/795e1c32-4def-4adb-abce-c118db86156f-config-volume\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.007670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-installation-pull-secrets\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzck\" (UniqueName: \"kubernetes.io/projected/795e1c32-4def-4adb-abce-c118db86156f-kube-api-access-fhzck\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.007670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-certificates\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795e1c32-4def-4adb-abce-c118db86156f-tmp-dir\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.007779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkjn\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-kube-api-access-4xkjn\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.007779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.007775    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-trusted-ca\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.048965 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.048929    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:23.049126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.049107    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:23.051845 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.051812    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 09:07:23.052238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.052037    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 09:07:23.052238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.052105    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fwjtr\""
Sep 15 09:07:23.090521 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.090491    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:23.108568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkjn\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-kube-api-access-4xkjn\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.108691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.108691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-trusted-ca\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.108691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdc7ebe9-46d4-4905-af95-2ae799a30157-cert\") pod \"ingress-canary-qwp25\" (UID: \"fdc7ebe9-46d4-4905-af95-2ae799a30157\") " pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cz9w\" (UniqueName: \"kubernetes.io/projected/fdc7ebe9-46d4-4905-af95-2ae799a30157-kube-api-access-5cz9w\") pod \"ingress-canary-qwp25\" (UID: \"fdc7ebe9-46d4-4905-af95-2ae799a30157\") " pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108716    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.108753    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.108770    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-785dcc4fcf-b6mzw: secret "image-registry-tls" not found
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.108810    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.108849    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls podName:dd2aced5-6d03-4660-9c20-cf2248046c9c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:23.608813621 +0000 UTC m=+36.068103139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls") pod "image-registry-785dcc4fcf-b6mzw" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c") : secret "image-registry-tls" not found
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.108877    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls podName:795e1c32-4def-4adb-abce-c118db86156f nodeName:}" failed. No retries permitted until 2026-09-15 09:07:23.608860786 +0000 UTC m=+36.068150319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls") pod "dns-default-vc4p7" (UID: "795e1c32-4def-4adb-abce-c118db86156f") : secret "dns-default-metrics-tls" not found
Sep 15 09:07:23.108875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-image-registry-private-configuration\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd2aced5-6d03-4660-9c20-cf2248046c9c-ca-trust-extracted\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-bound-sa-token\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.108985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/795e1c32-4def-4adb-abce-c118db86156f-config-volume\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-installation-pull-secrets\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzck\" (UniqueName: \"kubernetes.io/projected/795e1c32-4def-4adb-abce-c118db86156f-kube-api-access-fhzck\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-certificates\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795e1c32-4def-4adb-abce-c118db86156f-tmp-dir\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.109337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109235    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd2aced5-6d03-4660-9c20-cf2248046c9c-ca-trust-extracted\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109448    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/795e1c32-4def-4adb-abce-c118db86156f-tmp-dir\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.109733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/795e1c32-4def-4adb-abce-c118db86156f-config-volume\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.109733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109727    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-trusted-ca\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.109891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.109871    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-certificates\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.113071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.113040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-installation-pull-secrets\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.113071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.113054    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdc7ebe9-46d4-4905-af95-2ae799a30157-cert\") pod \"ingress-canary-qwp25\" (UID: \"fdc7ebe9-46d4-4905-af95-2ae799a30157\") " pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.113331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.113313    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-image-registry-private-configuration\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.117283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.117262    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzck\" (UniqueName: \"kubernetes.io/projected/795e1c32-4def-4adb-abce-c118db86156f-kube-api-access-fhzck\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.117588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.117567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cz9w\" (UniqueName: \"kubernetes.io/projected/fdc7ebe9-46d4-4905-af95-2ae799a30157-kube-api-access-5cz9w\") pod \"ingress-canary-qwp25\" (UID: \"fdc7ebe9-46d4-4905-af95-2ae799a30157\") " pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.118188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.118161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-bound-sa-token\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.118557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.118536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkjn\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-kube-api-access-4xkjn\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.192212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.192174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qwp25"
Sep 15 09:07:23.278298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.278215    2571 generic.go:356] "Generic (PLEG): container finished" podID="32565e69-75c9-48aa-a026-ed63498f2fa8" containerID="3878f055e6c3601b4208be977f39e9917aa06d74db393dbd3e6de392d9fe8e0e" exitCode=0
Sep 15 09:07:23.278298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.278269    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerDied","Data":"3878f055e6c3601b4208be977f39e9917aa06d74db393dbd3e6de392d9fe8e0e"}
Sep 15 09:07:23.512436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.512403    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qwp25"]
Sep 15 09:07:23.515292 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:23.515266    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc7ebe9_46d4_4905_af95_2ae799a30157.slice/crio-8ea8a12297e837dd493630704b5cd74963a66219c3f390b443ecb6b9d7c1a67b WatchSource:0}: Error finding container 8ea8a12297e837dd493630704b5cd74963a66219c3f390b443ecb6b9d7c1a67b: Status 404 returned error can't find the container with id 8ea8a12297e837dd493630704b5cd74963a66219c3f390b443ecb6b9d7c1a67b
Sep 15 09:07:23.611901 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.611870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:23.612065 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:23.611909    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:23.612065 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.612011    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:07:23.612065 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.612026    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:07:23.612065 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.612050    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-785dcc4fcf-b6mzw: secret "image-registry-tls" not found
Sep 15 09:07:23.612065 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.612063    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls podName:795e1c32-4def-4adb-abce-c118db86156f nodeName:}" failed. No retries permitted until 2026-09-15 09:07:24.61204876 +0000 UTC m=+37.071338278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls") pod "dns-default-vc4p7" (UID: "795e1c32-4def-4adb-abce-c118db86156f") : secret "dns-default-metrics-tls" not found
Sep 15 09:07:23.612245 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:23.612108    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls podName:dd2aced5-6d03-4660-9c20-cf2248046c9c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:24.612092759 +0000 UTC m=+37.071382278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls") pod "image-registry-785dcc4fcf-b6mzw" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c") : secret "image-registry-tls" not found
Sep 15 09:07:24.049010 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.048934    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:24.051838 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.051796    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 09:07:24.051975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.051797    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 09:07:24.051975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.051796    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxrmh\""
Sep 15 09:07:24.285374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.285338    2571 generic.go:356] "Generic (PLEG): container finished" podID="32565e69-75c9-48aa-a026-ed63498f2fa8" containerID="74f8995c3c9f722e17dfc723edd9f38c19fef4117d93916c10537355b8aba845" exitCode=0
Sep 15 09:07:24.285555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.285417    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerDied","Data":"74f8995c3c9f722e17dfc723edd9f38c19fef4117d93916c10537355b8aba845"}
Sep 15 09:07:24.286795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.286756    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qwp25" event={"ID":"fdc7ebe9-46d4-4905-af95-2ae799a30157","Type":"ContainerStarted","Data":"8ea8a12297e837dd493630704b5cd74963a66219c3f390b443ecb6b9d7c1a67b"}
Sep 15 09:07:24.575980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.575874    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:24.620513 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.620477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:24.620702 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.620530    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:24.620702 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:24.620638    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:07:24.620702 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:24.620655    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:07:24.620702 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:24.620674    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-785dcc4fcf-b6mzw: secret "image-registry-tls" not found
Sep 15 09:07:24.620702 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:24.620700    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls podName:795e1c32-4def-4adb-abce-c118db86156f nodeName:}" failed. No retries permitted until 2026-09-15 09:07:26.620685558 +0000 UTC m=+39.079975077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls") pod "dns-default-vc4p7" (UID: "795e1c32-4def-4adb-abce-c118db86156f") : secret "dns-default-metrics-tls" not found
Sep 15 09:07:24.620985 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:24.620719    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls podName:dd2aced5-6d03-4660-9c20-cf2248046c9c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:26.620712814 +0000 UTC m=+39.080002333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls") pod "image-registry-785dcc4fcf-b6mzw" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c") : secret "image-registry-tls" not found
Sep 15 09:07:24.676895 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:24.676808    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:25.293298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:25.293270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" event={"ID":"32565e69-75c9-48aa-a026-ed63498f2fa8","Type":"ContainerStarted","Data":"547cdbe42972150c8603cf1411d167ff2c16573e9b39c8fcdab3c7cef09081c5"}
Sep 15 09:07:25.320750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:25.320675    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgrjz" podStartSLOduration=6.03585319 podStartE2EDuration="36.320652747s" totalImagesPullingTime="30.284799557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:25.319422482 +0000 UTC m=+37.778712049" watchObservedRunningTime="2026-09-15 09:07:25.320652747 +0000 UTC m=+37.779942289"
Sep 15 09:07:25.442024 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:25.441968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:25.501292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:25.501243    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:26.641193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:26.641098    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:26.641193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:26.641147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:26.641666 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:26.641252    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:07:26.641666 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:26.641256    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:07:26.641666 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:26.641320    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-785dcc4fcf-b6mzw: secret "image-registry-tls" not found
Sep 15 09:07:26.641666 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:26.641324    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls podName:795e1c32-4def-4adb-abce-c118db86156f nodeName:}" failed. No retries permitted until 2026-09-15 09:07:30.641302955 +0000 UTC m=+43.100592487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls") pod "dns-default-vc4p7" (UID: "795e1c32-4def-4adb-abce-c118db86156f") : secret "dns-default-metrics-tls" not found
Sep 15 09:07:26.641666 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:26.641365    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls podName:dd2aced5-6d03-4660-9c20-cf2248046c9c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:30.641352815 +0000 UTC m=+43.100642334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls") pod "image-registry-785dcc4fcf-b6mzw" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c") : secret "image-registry-tls" not found
Sep 15 09:07:27.300267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:27.300232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qwp25" event={"ID":"fdc7ebe9-46d4-4905-af95-2ae799a30157","Type":"ContainerStarted","Data":"02a907e432a3ce03d1ab10a0123b3d14ec0d06e6cf59dd3603ac0f21535139ee"}
Sep 15 09:07:27.317081 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:27.317015    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qwp25" podStartSLOduration=2.483656678 podStartE2EDuration="5.316999706s" totalImagesPullingTime="2.833343028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:27.316161642 +0000 UTC m=+39.775451184" watchObservedRunningTime="2026-09-15 09:07:27.316999706 +0000 UTC m=+39.776289246"
Sep 15 09:07:27.855006 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:27.854975    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:27.912448 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:27.912392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:28.069675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:28.069642    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:28.129441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:28.129361    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:28.327262 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:28.327233    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:28.427370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:28.427281    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:29.065086 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:29.065049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:29.166250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:29.166221    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:30.671594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:30.671555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:30.671999 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:30.671670    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:30.671999 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:30.671728    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:07:30.671999 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:30.671752    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:07:30.671999 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:30.671763    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-785dcc4fcf-b6mzw: secret "image-registry-tls" not found
Sep 15 09:07:30.671999 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:30.671798    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls podName:795e1c32-4def-4adb-abce-c118db86156f nodeName:}" failed. No retries permitted until 2026-09-15 09:07:38.67178104 +0000 UTC m=+51.131070579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls") pod "dns-default-vc4p7" (UID: "795e1c32-4def-4adb-abce-c118db86156f") : secret "dns-default-metrics-tls" not found
Sep 15 09:07:30.671999 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:30.671814    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls podName:dd2aced5-6d03-4660-9c20-cf2248046c9c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:38.671808 +0000 UTC m=+51.131097519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls") pod "image-registry-785dcc4fcf-b6mzw" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c") : secret "image-registry-tls" not found
Sep 15 09:07:31.022982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.022910    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:31.208893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.208862    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"]
Sep 15 09:07:31.241305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.241270    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"]
Sep 15 09:07:31.241467 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.241386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.243774 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.243754    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 09:07:31.243924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.243792    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 09:07:31.244084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.244069    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9b4l9\""
Sep 15 09:07:31.255360 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.255337    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78c4fbf958-wdqdc"]
Sep 15 09:07:31.255459 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.255444    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.258008 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.257985    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 09:07:31.258119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.258020    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 09:07:31.258119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.258055    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 09:07:31.258119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.258057    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:07:31.258278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.258183    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r9qfw\""
Sep 15 09:07:31.273392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.273321    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"]
Sep 15 09:07:31.273392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.273355    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"]
Sep 15 09:07:31.273392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.273371    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78c4fbf958-wdqdc"]
Sep 15 09:07:31.273557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.273430    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.276124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276100    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 09:07:31.276257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276152    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vbm8q\""
Sep 15 09:07:31.276257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276181    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 09:07:31.276257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276202    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 09:07:31.276257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276151    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 09:07:31.276257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276157    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 09:07:31.276519 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.276498    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 09:07:31.310507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.310477    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-87nft"]
Sep 15 09:07:31.331400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.331374    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-87nft"]
Sep 15 09:07:31.331563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.331468    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.334123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.334094    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 09:07:31.334123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.334101    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 09:07:31.334357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.334343    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 09:07:31.334416 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.334397    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:07:31.334697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.334682    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qlt5s\""
Sep 15 09:07:31.339389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.339371    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 09:07:31.376865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.376809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.376865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.376864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.377080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.376922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.377080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.376957    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6cz\" (UniqueName: \"kubernetes.io/projected/f996fced-148b-4c99-9ad7-b885c09f618c-kube-api-access-rs6cz\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.377080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.376991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-kube-api-access-ttm5m\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.377080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.377015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-stats-auth\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.377080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.377058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.377230 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.377083    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.377230 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.377135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caa97be3-bc67-45b4-af76-1eab51a65a35-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.377230 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.377150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-default-certificate\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.458871 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.458842    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:31.478463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478431    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4492b-ba6e-4e83-9d63-6b89747de67e-trusted-ca\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.478600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.478600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.478682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caa97be3-bc67-45b4-af76-1eab51a65a35-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.478682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-default-certificate\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.478682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1d4492b-ba6e-4e83-9d63-6b89747de67e-config\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.478816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478690    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1d4492b-ba6e-4e83-9d63-6b89747de67e-serving-cert\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.478816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478792    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.478951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.478951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.478951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6cz\" (UniqueName: \"kubernetes.io/projected/f996fced-148b-4c99-9ad7-b885c09f618c-kube-api-access-rs6cz\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.478951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478911    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvld\" (UniqueName: \"kubernetes.io/projected/e1d4492b-ba6e-4e83-9d63-6b89747de67e-kube-api-access-6qvld\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.478951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-kube-api-access-ttm5m\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.479167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.478971    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-stats-auth\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.479167 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.478914    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:07:31.479167 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.478972    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 09:07:31.479167 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.478992    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:31.978973602 +0000 UTC m=+44.438263144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : configmap references non-existent config key: service-ca.crt
Sep 15 09:07:31.479167 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.479053    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:31.97903717 +0000 UTC m=+44.438326689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : secret "router-metrics-certs-default" not found
Sep 15 09:07:31.479408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.479210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.479408 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.479238    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert podName:caa97be3-bc67-45b4-af76-1eab51a65a35 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:31.979223324 +0000 UTC m=+44.438512850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fw5zp" (UID: "caa97be3-bc67-45b4-af76-1eab51a65a35") : secret "networking-console-plugin-cert" not found
Sep 15 09:07:31.479592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.479568    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/caa97be3-bc67-45b4-af76-1eab51a65a35-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.482088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.482065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.482194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.482140    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-stats-auth\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.482194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.482158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-default-certificate\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.489710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.489684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttm5m\" (UniqueName: \"kubernetes.io/projected/31b4dd4b-c91c-4e10-87cf-7dd643e2edee-kube-api-access-ttm5m\") pod \"kube-storage-version-migrator-operator-6c8f84f897-cj2td\" (UID: \"31b4dd4b-c91c-4e10-87cf-7dd643e2edee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.490019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.489996    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6cz\" (UniqueName: \"kubernetes.io/projected/f996fced-148b-4c99-9ad7-b885c09f618c-kube-api-access-rs6cz\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.557214 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.557140    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:31.564221 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.564195    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"
Sep 15 09:07:31.579711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.579683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1d4492b-ba6e-4e83-9d63-6b89747de67e-config\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.579785 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.579718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1d4492b-ba6e-4e83-9d63-6b89747de67e-serving-cert\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.579852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.579792    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvld\" (UniqueName: \"kubernetes.io/projected/e1d4492b-ba6e-4e83-9d63-6b89747de67e-kube-api-access-6qvld\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.579852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.579813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4492b-ba6e-4e83-9d63-6b89747de67e-trusted-ca\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.580416 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.580394    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1d4492b-ba6e-4e83-9d63-6b89747de67e-config\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.580912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.580893    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4492b-ba6e-4e83-9d63-6b89747de67e-trusted-ca\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.582108 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.582083    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e1d4492b-ba6e-4e83-9d63-6b89747de67e-serving-cert\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.588420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.588203    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvld\" (UniqueName: \"kubernetes.io/projected/e1d4492b-ba6e-4e83-9d63-6b89747de67e-kube-api-access-6qvld\") pod \"console-operator-696884fb9c-87nft\" (UID: \"e1d4492b-ba6e-4e83-9d63-6b89747de67e\") " pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.640212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.640184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:31.853893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.853861    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td"]
Sep 15 09:07:31.856382 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:31.856344    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b4dd4b_c91c_4e10_87cf_7dd643e2edee.slice/crio-0664e9f0efcf1b14949a5fdd262501967efd81301cee305e6f8b4892f2ce2fd6 WatchSource:0}: Error finding container 0664e9f0efcf1b14949a5fdd262501967efd81301cee305e6f8b4892f2ce2fd6: Status 404 returned error can't find the container with id 0664e9f0efcf1b14949a5fdd262501967efd81301cee305e6f8b4892f2ce2fd6
Sep 15 09:07:31.924045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.924010    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-87nft"]
Sep 15 09:07:31.926932 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:31.926905    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d4492b_ba6e_4e83_9d63_6b89747de67e.slice/crio-314c3470f2fac621010a40c9484a682efa828970bb076cb0778be490c81baae2 WatchSource:0}: Error finding container 314c3470f2fac621010a40c9484a682efa828970bb076cb0778be490c81baae2: Status 404 returned error can't find the container with id 314c3470f2fac621010a40c9484a682efa828970bb076cb0778be490c81baae2
Sep 15 09:07:31.983401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.983370    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:31.983401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.983405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.983613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:31.983423    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:31.983613 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.983517    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:07:31.983613 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.983560    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:32.983542085 +0000 UTC m=+45.442831621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : configmap references non-existent config key: service-ca.crt
Sep 15 09:07:31.983613 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.983579    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert podName:caa97be3-bc67-45b4-af76-1eab51a65a35 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:32.98357342 +0000 UTC m=+45.442862939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fw5zp" (UID: "caa97be3-bc67-45b4-af76-1eab51a65a35") : secret "networking-console-plugin-cert" not found
Sep 15 09:07:31.983613 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.983595    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 09:07:31.983777 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:31.983646    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:32.983631915 +0000 UTC m=+45.442921435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : secret "router-metrics-certs-default" not found
Sep 15 09:07:32.312224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.312133    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-87nft" event={"ID":"e1d4492b-ba6e-4e83-9d63-6b89747de67e","Type":"ContainerStarted","Data":"314c3470f2fac621010a40c9484a682efa828970bb076cb0778be490c81baae2"}
Sep 15 09:07:32.313173 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.313148    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td" event={"ID":"31b4dd4b-c91c-4e10-87cf-7dd643e2edee","Type":"ContainerStarted","Data":"0664e9f0efcf1b14949a5fdd262501967efd81301cee305e6f8b4892f2ce2fd6"}
Sep 15 09:07:32.378535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.378502    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:32.435527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.435494    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:32.505613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.505579    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:32.574818 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.574736    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:32.993508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.993454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.993517    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:32.993546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:32.993683    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:32.993683    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:32.993746    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:34.993727728 +0000 UTC m=+47.453017261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : secret "router-metrics-certs-default" not found
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:32.993763    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:34.993753528 +0000 UTC m=+47.453043054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : configmap references non-existent config key: service-ca.crt
Sep 15 09:07:32.994001 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:32.993777    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert podName:caa97be3-bc67-45b4-af76-1eab51a65a35 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:34.993769629 +0000 UTC m=+47.453059151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fw5zp" (UID: "caa97be3-bc67-45b4-af76-1eab51a65a35") : secret "networking-console-plugin-cert" not found
Sep 15 09:07:33.361277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:33.361232    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:35.008797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.008768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.008802    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.008822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:35.008946    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:35.008986    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:35.009009    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:39.008986197 +0000 UTC m=+51.468275736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : configmap references non-existent config key: service-ca.crt
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:35.009039    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert podName:caa97be3-bc67-45b4-af76-1eab51a65a35 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:39.009028281 +0000 UTC m=+51.468317805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fw5zp" (UID: "caa97be3-bc67-45b4-af76-1eab51a65a35") : secret "networking-console-plugin-cert" not found
Sep 15 09:07:35.009228 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:35.009056    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:39.00904638 +0000 UTC m=+51.468335899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : secret "router-metrics-certs-default" not found
Sep 15 09:07:35.323537 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.323454    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/0.log"
Sep 15 09:07:35.323537 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.323500    2571 generic.go:356] "Generic (PLEG): container finished" podID="e1d4492b-ba6e-4e83-9d63-6b89747de67e" containerID="d6a3523bcf118c4b37e211d3d9097611b4fdad0ff3866e005f50d976cb7e6c0d" exitCode=255
Sep 15 09:07:35.323753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.323581    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-87nft" event={"ID":"e1d4492b-ba6e-4e83-9d63-6b89747de67e","Type":"ContainerDied","Data":"d6a3523bcf118c4b37e211d3d9097611b4fdad0ff3866e005f50d976cb7e6c0d"}
Sep 15 09:07:35.323900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.323881    2571 scope.go:117] "RemoveContainer" containerID="d6a3523bcf118c4b37e211d3d9097611b4fdad0ff3866e005f50d976cb7e6c0d"
Sep 15 09:07:35.325283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.325254    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td" event={"ID":"31b4dd4b-c91c-4e10-87cf-7dd643e2edee","Type":"ContainerStarted","Data":"bab26ace3b6acaaa767ef53b9692ce480f44f70bee96a4b4e5a866ab57117896"}
Sep 15 09:07:35.356437 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:35.356388    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td" podStartSLOduration=1.363154399 podStartE2EDuration="4.356370454s" totalImagesPullingTime="2.993216055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:35.355980163 +0000 UTC m=+47.815269705" watchObservedRunningTime="2026-09-15 09:07:35.356370454 +0000 UTC m=+47.815659996"
Sep 15 09:07:36.069184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.069142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:36.079947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.079911    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"]
Sep 15 09:07:36.099838 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.099799    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"]
Sep 15 09:07:36.099970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.099938    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"
Sep 15 09:07:36.102629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.102601    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 09:07:36.103572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.103553    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mm6t8\""
Sep 15 09:07:36.103692 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.103559    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 09:07:36.117949 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.117924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9qj\" (UniqueName: \"kubernetes.io/projected/51c0546e-33df-4ae4-be9c-04e9a4dca2fc-kube-api-access-9n9qj\") pod \"migrator-8b4d697c5-dwb4h\" (UID: \"51c0546e-33df-4ae4-be9c-04e9a4dca2fc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"
Sep 15 09:07:36.143111 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.143082    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:36.210277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.210250    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:36.218430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.218406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9qj\" (UniqueName: \"kubernetes.io/projected/51c0546e-33df-4ae4-be9c-04e9a4dca2fc-kube-api-access-9n9qj\") pod \"migrator-8b4d697c5-dwb4h\" (UID: \"51c0546e-33df-4ae4-be9c-04e9a4dca2fc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"
Sep 15 09:07:36.226127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.226098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9qj\" (UniqueName: \"kubernetes.io/projected/51c0546e-33df-4ae4-be9c-04e9a4dca2fc-kube-api-access-9n9qj\") pod \"migrator-8b4d697c5-dwb4h\" (UID: \"51c0546e-33df-4ae4-be9c-04e9a4dca2fc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"
Sep 15 09:07:36.328984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.328910    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/1.log"
Sep 15 09:07:36.329364 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.329348    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/0.log"
Sep 15 09:07:36.329415 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.329387    2571 generic.go:356] "Generic (PLEG): container finished" podID="e1d4492b-ba6e-4e83-9d63-6b89747de67e" containerID="026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2" exitCode=255
Sep 15 09:07:36.329498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.329477    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-87nft" event={"ID":"e1d4492b-ba6e-4e83-9d63-6b89747de67e","Type":"ContainerDied","Data":"026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2"}
Sep 15 09:07:36.329568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.329556    2571 scope.go:117] "RemoveContainer" containerID="d6a3523bcf118c4b37e211d3d9097611b4fdad0ff3866e005f50d976cb7e6c0d"
Sep 15 09:07:36.329755 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.329734    2571 scope.go:117] "RemoveContainer" containerID="026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2"
Sep 15 09:07:36.329954 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:36.329924    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-87nft_openshift-console-operator(e1d4492b-ba6e-4e83-9d63-6b89747de67e)\"" pod="openshift-console-operator/console-operator-696884fb9c-87nft" podUID="e1d4492b-ba6e-4e83-9d63-6b89747de67e"
Sep 15 09:07:36.409792 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.409752    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"
Sep 15 09:07:36.699937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:36.699904    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h"]
Sep 15 09:07:36.702872 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:36.702814    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c0546e_33df_4ae4_be9c_04e9a4dca2fc.slice/crio-865d36cfb74692764e84cf02105faebf4f6a6e48ff77175ce80b82459865328f WatchSource:0}: Error finding container 865d36cfb74692764e84cf02105faebf4f6a6e48ff77175ce80b82459865328f: Status 404 returned error can't find the container with id 865d36cfb74692764e84cf02105faebf4f6a6e48ff77175ce80b82459865328f
Sep 15 09:07:37.333858 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.333805    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h" event={"ID":"51c0546e-33df-4ae4-be9c-04e9a4dca2fc","Type":"ContainerStarted","Data":"865d36cfb74692764e84cf02105faebf4f6a6e48ff77175ce80b82459865328f"}
Sep 15 09:07:37.335837 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.335805    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/1.log"
Sep 15 09:07:37.336171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.336148    2571 scope.go:117] "RemoveContainer" containerID="026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2"
Sep 15 09:07:37.336381 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:37.336357    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-87nft_openshift-console-operator(e1d4492b-ba6e-4e83-9d63-6b89747de67e)\"" pod="openshift-console-operator/console-operator-696884fb9c-87nft" podUID="e1d4492b-ba6e-4e83-9d63-6b89747de67e"
Sep 15 09:07:37.388479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.388439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:37.469659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.469621    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:37.699659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.699568    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wfwmf"]
Sep 15 09:07:37.715863 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.715812    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:37.716974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.716948    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wfwmf"]
Sep 15 09:07:37.717098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.717022    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.719691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.719610    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 09:07:37.719691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.719634    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 09:07:37.719691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.719665    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 09:07:37.719935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.719666    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 09:07:37.720815 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.720789    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rwr78\""
Sep 15 09:07:37.733099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.733002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.733099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.733047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-crio-socket\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.733099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.733076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.733510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.733175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-data-volume\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.733510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.733224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffm2j\" (UniqueName: \"kubernetes.io/projected/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-api-access-ffm2j\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.781841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.781803    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:37.834037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.833995    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.834208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.834068    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-data-volume\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.834208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.834106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffm2j\" (UniqueName: \"kubernetes.io/projected/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-api-access-ffm2j\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.834208 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:37.834123    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:07:37.834208 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:37.834188    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:38.334168782 +0000 UTC m=+50.793458316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : configmap "kube-rbac-proxy" not found
Sep 15 09:07:37.834374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.834262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.834374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.834294    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-crio-socket\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.834443 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:37.834369    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:37.834443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.834379    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-data-volume\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.834443 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:37.834431    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:38.334414011 +0000 UTC m=+50.793703533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:37.834572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.834453    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-crio-socket\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:37.842542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:37.842513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffm2j\" (UniqueName: \"kubernetes.io/projected/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-api-access-ffm2j\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:38.338501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:38.338477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:38.338853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:38.338509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:38.338853 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.338624    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 09:07:38.338853 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.338662    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:38.338853 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.338682    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:39.338664631 +0000 UTC m=+51.797954166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : configmap "kube-rbac-proxy" not found
Sep 15 09:07:38.338853 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.338723    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:39.338706258 +0000 UTC m=+51.797995799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:38.340428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:38.340398    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h" event={"ID":"51c0546e-33df-4ae4-be9c-04e9a4dca2fc","Type":"ContainerStarted","Data":"2196fabfffa0ebfbc478f2ef11860fafbebaf7f3a1f717941a6b052d7f4c5cf2"}
Sep 15 09:07:38.742512 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:38.742476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:38.742673 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:38.742528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:38.742673 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.742616    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 09:07:38.742673 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.742622    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 09:07:38.742673 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.742639    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-785dcc4fcf-b6mzw: secret "image-registry-tls" not found
Sep 15 09:07:38.742673 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.742672    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls podName:795e1c32-4def-4adb-abce-c118db86156f nodeName:}" failed. No retries permitted until 2026-09-15 09:07:54.742660055 +0000 UTC m=+67.201949580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls") pod "dns-default-vc4p7" (UID: "795e1c32-4def-4adb-abce-c118db86156f") : secret "dns-default-metrics-tls" not found
Sep 15 09:07:38.742861 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:38.742685    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls podName:dd2aced5-6d03-4660-9c20-cf2248046c9c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:54.742679425 +0000 UTC m=+67.201968944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls") pod "image-registry-785dcc4fcf-b6mzw" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c") : secret "image-registry-tls" not found
Sep 15 09:07:38.955126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:38.955093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:39.013614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.013528    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:39.045933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.045904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:39.046030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.045942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:39.046030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.045989    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:39.046108 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.046049    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 09:07:39.046145 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.046122    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:47.046103646 +0000 UTC m=+59.505393200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : secret "router-metrics-certs-default" not found
Sep 15 09:07:39.046145 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.046136    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:07:39.046224 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.046151    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:07:47.046135981 +0000 UTC m=+59.505425527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : configmap references non-existent config key: service-ca.crt
Sep 15 09:07:39.046224 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.046170    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert podName:caa97be3-bc67-45b4-af76-1eab51a65a35 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:47.046164243 +0000 UTC m=+59.505453762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fw5zp" (UID: "caa97be3-bc67-45b4-af76-1eab51a65a35") : secret "networking-console-plugin-cert" not found
Sep 15 09:07:39.212148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.212117    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:39.344450 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.344413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h" event={"ID":"51c0546e-33df-4ae4-be9c-04e9a4dca2fc","Type":"ContainerStarted","Data":"1628ab92eadbaded9a97036fc7502af5cf64c644323fb57321e6becfd7b8ae06"}
Sep 15 09:07:39.348785 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.348765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:39.348907 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.348794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:39.348948 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.348912    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:39.348981 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:39.348970    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:41.348956213 +0000 UTC m=+53.808245732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:39.349246 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.349229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:39.452107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.452078    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:39.553433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.553403    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:39.657799 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.657717    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:39.888148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:39.888118    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxw6x_ef628e49-f57f-43ab-a6df-2eed9e923881/dns-node-resolver/0.log"
Sep 15 09:07:40.222980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:40.222947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:40.280981 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:40.280948    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:40.831575 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:40.831513    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:40.887334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:40.887311    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2494n_10e5b976-0d90-4ed2-a797-274776807382/node-ca/0.log"
Sep 15 09:07:40.930724 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:40.930698    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:41.365443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:41.365403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:41.365610 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:41.365553    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:41.365649 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:41.365612    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:45.36559779 +0000 UTC m=+57.824887329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:41.640574 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:41.640489    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:41.640574 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:41.640555    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:07:41.640924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:41.640911    2571 scope.go:117] "RemoveContainer" containerID="026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2"
Sep 15 09:07:41.641072 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:41.641053    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-87nft_openshift-console-operator(e1d4492b-ba6e-4e83-9d63-6b89747de67e)\"" pod="openshift-console-operator/console-operator-696884fb9c-87nft" podUID="e1d4492b-ba6e-4e83-9d63-6b89747de67e"
Sep 15 09:07:41.687712 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:41.687684    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qwp25_fdc7ebe9-46d4-4905-af95-2ae799a30157/serve-healthcheck-canary/0.log"
Sep 15 09:07:41.691619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:41.691595    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:42.088839 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:42.088808    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-dwb4h_51c0546e-33df-4ae4-be9c-04e9a4dca2fc/migrator/0.log"
Sep 15 09:07:42.288425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:42.288387    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-dwb4h_51c0546e-33df-4ae4-be9c-04e9a4dca2fc/graceful-termination/0.log"
Sep 15 09:07:42.488738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:42.488651    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-cj2td_31b4dd4b-c91c-4e10-87cf-7dd643e2edee/kube-storage-version-migrator-operator/0.log"
Sep 15 09:07:44.290093 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:44.290067    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:45.399952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:45.399821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:45.400395 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:45.399967    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:45.400395 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:45.400088    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls podName:0a59c6c9-0815-4d7e-9e51-32da9d62bbd2 nodeName:}" failed. No retries permitted until 2026-09-15 09:07:53.400063836 +0000 UTC m=+65.859353369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wfwmf" (UID: "0a59c6c9-0815-4d7e-9e51-32da9d62bbd2") : secret "insights-runtime-extractor-tls" not found
Sep 15 09:07:45.512800 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:45.512768    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:46.713325 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:46.713295    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:46.813692 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:46.813661    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:46.913313 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:46.913283    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:47.015209 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.015139    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:47.111997 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.111959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:07:47.112147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.112054    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:47.112147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.112074    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:07:47.112147 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:47.112107    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 15 09:07:47.112243 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:47.112185    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert podName:caa97be3-bc67-45b4-af76-1eab51a65a35 nodeName:}" failed. No retries permitted until 2026-09-15 09:08:03.112169691 +0000 UTC m=+75.571459233 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-fw5zp" (UID: "caa97be3-bc67-45b4-af76-1eab51a65a35") : secret "networking-console-plugin-cert" not found
Sep 15 09:07:47.112243 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:47.112186    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 09:07:47.112243 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:47.112207    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:08:03.112196198 +0000 UTC m=+75.571485716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : configmap references non-existent config key: service-ca.crt
Sep 15 09:07:47.112243 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:47.112221    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs podName:f996fced-148b-4c99-9ad7-b885c09f618c nodeName:}" failed. No retries permitted until 2026-09-15 09:08:03.112214987 +0000 UTC m=+75.571504506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs") pod "router-default-78c4fbf958-wdqdc" (UID: "f996fced-148b-4c99-9ad7-b885c09f618c") : secret "router-metrics-certs-default" not found
Sep 15 09:07:47.157098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.157066    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:47.276647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.276567    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-89jmt"
Sep 15 09:07:47.280807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.280776    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:47.318534 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.318468    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dwb4h" podStartSLOduration=9.828540524 podStartE2EDuration="11.318448594s" totalImagesPullingTime="1.48990807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:39.359856357 +0000 UTC m=+51.819145889" watchObservedRunningTime="2026-09-15 09:07:47.318448594 +0000 UTC m=+59.777738136"
Sep 15 09:07:47.418478 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:47.418447    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:48.620353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:48.620316    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:48.814439 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:48.814391    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:49.016861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:49.016760    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:49.223564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:49.223523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:49.419385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:49.419349    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:50.151015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:50.150972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:50.252891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:50.252848    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:50.411645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:50.411552    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:50.824682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:50.824569    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:51.013914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:51.013846    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:51.068758 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:51.068691    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:51.217807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:51.217770    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:51.685892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:51.685853    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:52.219015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:52.218979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:52.756977 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:52.756947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:52.858841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:52.858809    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:52.964213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:52.964178    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:53.066366 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.066289    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:53.256906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.256863    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:53.421969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.421918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:53.464612 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.464578    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:53.466822 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.466794    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a59c6c9-0815-4d7e-9e51-32da9d62bbd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wfwmf\" (UID: \"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2\") " pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:53.632107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.632074    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rwr78\""
Sep 15 09:07:53.639865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.639820    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wfwmf"
Sep 15 09:07:53.767891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.767848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:07:53.768050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.767905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:53.770744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.770727    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 09:07:53.770795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.770756    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 09:07:53.778943 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:53.778923    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 09:07:53.779035 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:53.778982    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs podName:84527138-dafc-4435-b121-9215368bcb16 nodeName:}" failed. No retries permitted until 2026-09-15 09:08:57.778966247 +0000 UTC m=+130.238255779 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs") pod "network-metrics-daemon-d8h8m" (UID: "84527138-dafc-4435-b121-9215368bcb16") : secret "metrics-daemon-secret" not found
Sep 15 09:07:53.781108 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.781080    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/22a36c08-dd46-4c1c-85a6-d53a59b56d41-original-pull-secret\") pod \"global-pull-secret-syncer-dqkpv\" (UID: \"22a36c08-dd46-4c1c-85a6-d53a59b56d41\") " pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:53.940011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.939939    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wfwmf"]
Sep 15 09:07:53.942853 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:53.942809    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a59c6c9_0815_4d7e_9e51_32da9d62bbd2.slice/crio-456975013e979022626c70489fb68ed45bf353c558e2d5a3bb17687abe715468 WatchSource:0}: Error finding container 456975013e979022626c70489fb68ed45bf353c558e2d5a3bb17687abe715468: Status 404 returned error can't find the container with id 456975013e979022626c70489fb68ed45bf353c558e2d5a3bb17687abe715468
Sep 15 09:07:53.963567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.963539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqkpv"
Sep 15 09:07:53.969928 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.969901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:53.972384 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.972364    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 09:07:53.983409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.983386    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 09:07:53.994121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:53.994009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsljb\" (UniqueName: \"kubernetes.io/projected/7580fb52-d812-428d-b561-f5cd4e97fcf4-kube-api-access-fsljb\") pod \"network-check-target-xd2s6\" (UID: \"7580fb52-d812-428d-b561-f5cd4e97fcf4\") " pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:54.049721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.049699    2571 scope.go:117] "RemoveContainer" containerID="026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2"
Sep 15 09:07:54.061401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.061376    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxrmh\""
Sep 15 09:07:54.068977 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.068957    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:07:54.257568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.257534    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dqkpv"]
Sep 15 09:07:54.261334 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:54.261309    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a36c08_dd46_4c1c_85a6_d53a59b56d41.slice/crio-bf404112d1ef74471e22817cffc3d4d3272ac2be906199d7be92c09452637e3f WatchSource:0}: Error finding container bf404112d1ef74471e22817cffc3d4d3272ac2be906199d7be92c09452637e3f: Status 404 returned error can't find the container with id bf404112d1ef74471e22817cffc3d4d3272ac2be906199d7be92c09452637e3f
Sep 15 09:07:54.357548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.357517    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xd2s6"]
Sep 15 09:07:54.360693 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:54.360663    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7580fb52_d812_428d_b561_f5cd4e97fcf4.slice/crio-63e5b890170aca21f6c88cbb68daaa3707907bfdfc5eef534b2974449bfbf7e0 WatchSource:0}: Error finding container 63e5b890170aca21f6c88cbb68daaa3707907bfdfc5eef534b2974449bfbf7e0: Status 404 returned error can't find the container with id 63e5b890170aca21f6c88cbb68daaa3707907bfdfc5eef534b2974449bfbf7e0
Sep 15 09:07:54.386314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.386281    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:07:54.386750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.386734    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/1.log"
Sep 15 09:07:54.386857 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.386766    2571 generic.go:356] "Generic (PLEG): container finished" podID="e1d4492b-ba6e-4e83-9d63-6b89747de67e" containerID="d4592433e367b1b8f8a99bc4d692ac3c093897fa87fc6cf1f1ef80a5630a86c3" exitCode=255
Sep 15 09:07:54.386916 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.386860    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-87nft" event={"ID":"e1d4492b-ba6e-4e83-9d63-6b89747de67e","Type":"ContainerDied","Data":"d4592433e367b1b8f8a99bc4d692ac3c093897fa87fc6cf1f1ef80a5630a86c3"}
Sep 15 09:07:54.386964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.386923    2571 scope.go:117] "RemoveContainer" containerID="026b722da777c8a44923ebc4d72b561a3da9f546c004db0cfab107713e5f60e2"
Sep 15 09:07:54.387268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.387253    2571 scope.go:117] "RemoveContainer" containerID="d4592433e367b1b8f8a99bc4d692ac3c093897fa87fc6cf1f1ef80a5630a86c3"
Sep 15 09:07:54.387469 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:07:54.387439    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-87nft_openshift-console-operator(e1d4492b-ba6e-4e83-9d63-6b89747de67e)\"" pod="openshift-console-operator/console-operator-696884fb9c-87nft" podUID="e1d4492b-ba6e-4e83-9d63-6b89747de67e"
Sep 15 09:07:54.388389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.388360    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xd2s6" event={"ID":"7580fb52-d812-428d-b561-f5cd4e97fcf4","Type":"ContainerStarted","Data":"63e5b890170aca21f6c88cbb68daaa3707907bfdfc5eef534b2974449bfbf7e0"}
Sep 15 09:07:54.389481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.389460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dqkpv" event={"ID":"22a36c08-dd46-4c1c-85a6-d53a59b56d41","Type":"ContainerStarted","Data":"bf404112d1ef74471e22817cffc3d4d3272ac2be906199d7be92c09452637e3f"}
Sep 15 09:07:54.390880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.390857    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wfwmf" event={"ID":"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2","Type":"ContainerStarted","Data":"a441b0011ffe3c39b772ba9354ff42d830c5eab14403e95c8219362d9cc3a36c"}
Sep 15 09:07:54.390953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.390908    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wfwmf" event={"ID":"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2","Type":"ContainerStarted","Data":"456975013e979022626c70489fb68ed45bf353c558e2d5a3bb17687abe715468"}
Sep 15 09:07:54.417228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.417206    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:54.611102 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.611061    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:54.661901 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.661867    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:54.777768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.777729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:54.777939 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.777820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:54.780244 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.780219    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/795e1c32-4def-4adb-abce-c118db86156f-metrics-tls\") pod \"dns-default-vc4p7\" (UID: \"795e1c32-4def-4adb-abce-c118db86156f\") " pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:54.780422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.780406    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"image-registry-785dcc4fcf-b6mzw\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") " pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:54.963490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.963451    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8vjg9\""
Sep 15 09:07:54.971668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.971643    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:54.975135 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.975113    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lb4vr\""
Sep 15 09:07:54.983675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:54.983648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:07:55.182393 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.182310    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:55.261043 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.260954    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-785dcc4fcf-b6mzw"]
Sep 15 09:07:55.265698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.265600    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:55.268047 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.267778    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vc4p7"]
Sep 15 09:07:55.271043 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:07:55.270985    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795e1c32_4def_4adb_abce_c118db86156f.slice/crio-1bedcecb7ffe74cf8de8fa3dda7b23812840b63abfc022a185f77256ad46a778 WatchSource:0}: Error finding container 1bedcecb7ffe74cf8de8fa3dda7b23812840b63abfc022a185f77256ad46a778: Status 404 returned error can't find the container with id 1bedcecb7ffe74cf8de8fa3dda7b23812840b63abfc022a185f77256ad46a778
Sep 15 09:07:55.404398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.404232    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:07:55.407465 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.407432    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wfwmf" event={"ID":"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2","Type":"ContainerStarted","Data":"f88184be486fb860b5b5da4c7d1ce5b91b2b4a9d31dda7e150c8901b4a3e98c9"}
Sep 15 09:07:55.409013 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.408987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vc4p7" event={"ID":"795e1c32-4def-4adb-abce-c118db86156f","Type":"ContainerStarted","Data":"1bedcecb7ffe74cf8de8fa3dda7b23812840b63abfc022a185f77256ad46a778"}
Sep 15 09:07:55.411121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.411096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" event={"ID":"dd2aced5-6d03-4660-9c20-cf2248046c9c","Type":"ContainerStarted","Data":"63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53"}
Sep 15 09:07:55.411282 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.411261    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" event={"ID":"dd2aced5-6d03-4660-9c20-cf2248046c9c","Type":"ContainerStarted","Data":"4752c0ae3d4cc96bcf06fc7a801a68b3353ea0019304edfdf623d38ecfc59cd8"}
Sep 15 09:07:55.412209 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.412189    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:07:55.431745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.431688    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" podStartSLOduration=66.431668299 podStartE2EDuration="1m6.431668299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:55.431049705 +0000 UTC m=+67.890339247" watchObservedRunningTime="2026-09-15 09:07:55.431668299 +0000 UTC m=+67.890957842"
Sep 15 09:07:55.621814 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:55.621776    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:56.419879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:56.419817    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:56.818937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:56.818806    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:57.018851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:57.018785    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:57.219211 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:57.219171    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:57.419180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:57.419150    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:07:57.422105 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:57.422074    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wfwmf" event={"ID":"0a59c6c9-0815-4d7e-9e51-32da9d62bbd2","Type":"ContainerStarted","Data":"12dd8bee103d51e533427a310658f513aa7f01ab3c6ca6720943b2feefb4aee4"}
Sep 15 09:07:57.440049 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:07:57.439989    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wfwmf" podStartSLOduration=18.045517192 podStartE2EDuration="20.439965422s" totalImagesPullingTime="2.39444823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:07:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:07:57.439182214 +0000 UTC m=+69.898471756" watchObservedRunningTime="2026-09-15 09:07:57.439965422 +0000 UTC m=+69.899254966"
Sep 15 09:08:00.433975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.433935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xd2s6" event={"ID":"7580fb52-d812-428d-b561-f5cd4e97fcf4","Type":"ContainerStarted","Data":"95b4f0be6adda711aa8becb48ba6b34c9344dd6bc0e5f3333d3643a0ca151194"}
Sep 15 09:08:00.434496 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.434024    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:08:00.435479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.435453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dqkpv" event={"ID":"22a36c08-dd46-4c1c-85a6-d53a59b56d41","Type":"ContainerStarted","Data":"992309725eec0ebcfb44ffce45121e8ecad0404670278bbc3452411b4108a9b7"}
Sep 15 09:08:00.437310 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.437286    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vc4p7" event={"ID":"795e1c32-4def-4adb-abce-c118db86156f","Type":"ContainerStarted","Data":"d8b478adb0a849fd95e5360bdc459dc86a128a4bd5e40e08312138886a60c422"}
Sep 15 09:08:00.437414 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.437329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vc4p7" event={"ID":"795e1c32-4def-4adb-abce-c118db86156f","Type":"ContainerStarted","Data":"b708681d18025ef19f30b9f1e96356a715156bfdd18f27ab66c104483b720444"}
Sep 15 09:08:00.437414 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.437352    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:08:00.449794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.449750    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xd2s6" podStartSLOduration=66.139420441 podStartE2EDuration="1m11.449734963s" totalImagesPullingTime="5.310314522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:00.449086253 +0000 UTC m=+72.908375793" watchObservedRunningTime="2026-09-15 09:08:00.449734963 +0000 UTC m=+72.909024503"
Sep 15 09:08:00.465706 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.465658    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vc4p7" podStartSLOduration=34.073000558 podStartE2EDuration="38.465644053s" totalImagesPullingTime="4.392643495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:00.464856385 +0000 UTC m=+72.924145925" watchObservedRunningTime="2026-09-15 09:08:00.465644053 +0000 UTC m=+72.924933594"
Sep 15 09:08:00.480422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:00.480381    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dqkpv" podStartSLOduration=66.066425234 podStartE2EDuration="1m11.480367063s" totalImagesPullingTime="5.413941829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:00.479529639 +0000 UTC m=+72.938819180" watchObservedRunningTime="2026-09-15 09:08:00.480367063 +0000 UTC m=+72.939656603"
Sep 15 09:08:01.640631 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:01.640596    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:08:01.641044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:01.640654    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:08:01.641044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:01.640952    2571 scope.go:117] "RemoveContainer" containerID="d4592433e367b1b8f8a99bc4d692ac3c093897fa87fc6cf1f1ef80a5630a86c3"
Sep 15 09:08:01.641113 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:08:01.641092    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-87nft_openshift-console-operator(e1d4492b-ba6e-4e83-9d63-6b89747de67e)\"" pod="openshift-console-operator/console-operator-696884fb9c-87nft" podUID="e1d4492b-ba6e-4e83-9d63-6b89747de67e"
Sep 15 09:08:03.149914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.149875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:08:03.150400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.149944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:03.150400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.149965    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:03.150497 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.150479    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f996fced-148b-4c99-9ad7-b885c09f618c-service-ca-bundle\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:03.152285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.152262    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/caa97be3-bc67-45b4-af76-1eab51a65a35-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-fw5zp\" (UID: \"caa97be3-bc67-45b4-af76-1eab51a65a35\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:08:03.152285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.152273    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f996fced-148b-4c99-9ad7-b885c09f618c-metrics-certs\") pod \"router-default-78c4fbf958-wdqdc\" (UID: \"f996fced-148b-4c99-9ad7-b885c09f618c\") " pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:03.352303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.352270    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9b4l9\""
Sep 15 09:08:03.360127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.360108    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"
Sep 15 09:08:03.384423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.384398    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vbm8q\""
Sep 15 09:08:03.392463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.392434    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:03.635062 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.635028    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp"]
Sep 15 09:08:03.637874 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:03.637849    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa97be3_bc67_45b4_af76_1eab51a65a35.slice/crio-6197fdf50cb6abe4d486b08d988759cb289e457006bae241568417eab4a17f78 WatchSource:0}: Error finding container 6197fdf50cb6abe4d486b08d988759cb289e457006bae241568417eab4a17f78: Status 404 returned error can't find the container with id 6197fdf50cb6abe4d486b08d988759cb289e457006bae241568417eab4a17f78
Sep 15 09:08:03.697292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.697267    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78c4fbf958-wdqdc"]
Sep 15 09:08:03.699616 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:03.699587    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf996fced_148b_4c99_9ad7_b885c09f618c.slice/crio-6fd144e67f44635cba7c2f82bfa11626b4d9ba846325f2a7110b04377763b44e WatchSource:0}: Error finding container 6fd144e67f44635cba7c2f82bfa11626b4d9ba846325f2a7110b04377763b44e: Status 404 returned error can't find the container with id 6fd144e67f44635cba7c2f82bfa11626b4d9ba846325f2a7110b04377763b44e
Sep 15 09:08:03.951270 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:03.951195    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:04.046572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.046536    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:04.146419 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.146383    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:04.247462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.247378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:04.457054 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.457018    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp" event={"ID":"caa97be3-bc67-45b4-af76-1eab51a65a35","Type":"ContainerStarted","Data":"6197fdf50cb6abe4d486b08d988759cb289e457006bae241568417eab4a17f78"}
Sep 15 09:08:04.458672 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.458641    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78c4fbf958-wdqdc" event={"ID":"f996fced-148b-4c99-9ad7-b885c09f618c","Type":"ContainerStarted","Data":"dcf49ee724e3c936db172828b0eeb57626b899c487d3ddc553c1407459a8c05d"}
Sep 15 09:08:04.458812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.458688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78c4fbf958-wdqdc" event={"ID":"f996fced-148b-4c99-9ad7-b885c09f618c","Type":"ContainerStarted","Data":"6fd144e67f44635cba7c2f82bfa11626b4d9ba846325f2a7110b04377763b44e"}
Sep 15 09:08:04.477249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:04.477037    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78c4fbf958-wdqdc" podStartSLOduration=33.477015244 podStartE2EDuration="33.477015244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:04.476592357 +0000 UTC m=+76.935881884" watchObservedRunningTime="2026-09-15 09:08:04.477015244 +0000 UTC m=+76.936304787"
Sep 15 09:08:05.393324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.393287    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:05.395627 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.395605    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:05.463779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.463743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp" event={"ID":"caa97be3-bc67-45b4-af76-1eab51a65a35","Type":"ContainerStarted","Data":"1356a546181bfefdf3f0084681d9bc69c51e1ec4308201cdd5b446a7615e66db"}
Sep 15 09:08:05.464001 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.463980    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:05.465099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.465078    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78c4fbf958-wdqdc"
Sep 15 09:08:05.479319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.479270    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-fw5zp" podStartSLOduration=33.462552189 podStartE2EDuration="34.479256305s" totalImagesPullingTime="1.016704116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:05.47828585 +0000 UTC m=+77.937575391" watchObservedRunningTime="2026-09-15 09:08:05.479256305 +0000 UTC m=+77.938545861"
Sep 15 09:08:05.614324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.614287    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:05.668993 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:05.668918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:06.111549 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:06.111520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:06.168382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:06.168350    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:06.468648 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:06.468576    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:06.567320 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:06.567290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:06.955337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:06.955304    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:07.048259 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:07.048227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:07.126343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:07.126306    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:07.508239 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:07.508184    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:09.571986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:09.571951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:09.631158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:09.631121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:10.442211 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:10.442087    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vc4p7"
Sep 15 09:08:11.137975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:11.137938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:11.205977 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:11.205947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:11.571585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:11.571501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:13.234748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:13.234717    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:13.297940 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:13.297908    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:14.976407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:14.976358    2571 patch_prober.go:28] interesting pod/image-registry-785dcc4fcf-b6mzw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 09:08:14.976884 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:14.976421    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" podUID="dd2aced5-6d03-4660-9c20-cf2248046c9c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 09:08:15.046001 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:15.045961    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:17.049307 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.049268    2571 scope.go:117] "RemoveContainer" containerID="d4592433e367b1b8f8a99bc4d692ac3c093897fa87fc6cf1f1ef80a5630a86c3"
Sep 15 09:08:17.426744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.426715    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:08:17.513444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.513275    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:08:17.513444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.513352    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-87nft" event={"ID":"e1d4492b-ba6e-4e83-9d63-6b89747de67e","Type":"ContainerStarted","Data":"76051756908b2c21e4f53dadbe1199eba33eac3a63d021535b46847dc2820077"}
Sep 15 09:08:17.514092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.514063    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:08:17.514759 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.514735    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2cw7w"]
Sep 15 09:08:17.518379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.518360    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.521182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521159    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 09:08:17.521342 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521326    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 09:08:17.521417 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521372    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 09:08:17.521417 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521407    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2gqnx\""
Sep 15 09:08:17.521593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521418    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 09:08:17.521593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521485    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 09:08:17.521774 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.521760    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 09:08:17.527438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.527414    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"]
Sep 15 09:08:17.532087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.532063    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.535311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.535288    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 09:08:17.535443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.535288    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xnlc7\""
Sep 15 09:08:17.535443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.535296    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 09:08:17.535443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.535338    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 09:08:17.540346 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.540304    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-87nft" podStartSLOduration=43.612931355 podStartE2EDuration="46.540289404s" totalImagesPullingTime="2.927358049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:17.538367843 +0000 UTC m=+89.997657385" watchObservedRunningTime="2026-09-15 09:08:17.540289404 +0000 UTC m=+89.999579018"
Sep 15 09:08:17.547288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.547266    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"]
Sep 15 09:08:17.663764 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.663975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.663975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-wtmp\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.663975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.663975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663915    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdq4v\" (UniqueName: \"kubernetes.io/projected/33113bd9-0d82-42ea-925a-1150385869eb-kube-api-access-kdq4v\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.663975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663964    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.664182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.663985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33113bd9-0d82-42ea-925a-1150385869eb-metrics-client-ca\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.664182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f00af179-f92d-4591-a7fe-41397aa3c5b1-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.664182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f00af179-f92d-4591-a7fe-41397aa3c5b1-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.664182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-tls\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.664182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-sys\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.664334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664214    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtz2d\" (UniqueName: \"kubernetes.io/projected/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-api-access-qtz2d\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.664334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.664334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664308    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-root\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.664453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.664332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-textfile\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.765277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.765277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-root\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.765277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-textfile\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.765552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765340    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.765552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765380    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.765552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-wtmp\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.765552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.765552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-root\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.765552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdq4v\" (UniqueName: \"kubernetes.io/projected/33113bd9-0d82-42ea-925a-1150385869eb-kube-api-access-kdq4v\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765574    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33113bd9-0d82-42ea-925a-1150385869eb-metrics-client-ca\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f00af179-f92d-4591-a7fe-41397aa3c5b1-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f00af179-f92d-4591-a7fe-41397aa3c5b1-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-tls\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-sys\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtz2d\" (UniqueName: \"kubernetes.io/projected/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-api-access-qtz2d\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.766365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766039    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.766365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766175    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f00af179-f92d-4591-a7fe-41397aa3c5b1-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.766365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.765668    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-textfile\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766216    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-accelerators-collector-config\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-sys\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766365 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:08:17.766325    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 09:08:17.766657 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:08:17.766382    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-tls podName:33113bd9-0d82-42ea-925a-1150385869eb nodeName:}" failed. No retries permitted until 2026-09-15 09:08:18.266364322 +0000 UTC m=+90.725653857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-tls") pod "node-exporter-2cw7w" (UID: "33113bd9-0d82-42ea-925a-1150385869eb") : secret "node-exporter-tls" not found
Sep 15 09:08:17.766657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766582    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-wtmp\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766655    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33113bd9-0d82-42ea-925a-1150385869eb-metrics-client-ca\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.766988 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.766967    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f00af179-f92d-4591-a7fe-41397aa3c5b1-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.768416 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.768397    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.768509 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.768487    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.768552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.768488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.773436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.773412    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtz2d\" (UniqueName: \"kubernetes.io/projected/f00af179-f92d-4591-a7fe-41397aa3c5b1-kube-api-access-qtz2d\") pod \"kube-state-metrics-59fbdd77d-fn6dl\" (UID: \"f00af179-f92d-4591-a7fe-41397aa3c5b1\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:17.774087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.774068    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdq4v\" (UniqueName: \"kubernetes.io/projected/33113bd9-0d82-42ea-925a-1150385869eb-kube-api-access-kdq4v\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:17.842357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:17.842320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"
Sep 15 09:08:18.124064 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.123985    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl"]
Sep 15 09:08:18.126262 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:18.126233    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00af179_f92d_4591_a7fe_41397aa3c5b1.slice/crio-a2d5385eee1d037b699acb5fc6c05711ba8bf1f40d5ae256cdcaae3cb4ca7da6 WatchSource:0}: Error finding container a2d5385eee1d037b699acb5fc6c05711ba8bf1f40d5ae256cdcaae3cb4ca7da6: Status 404 returned error can't find the container with id a2d5385eee1d037b699acb5fc6c05711ba8bf1f40d5ae256cdcaae3cb4ca7da6
Sep 15 09:08:18.232783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.232749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:18.270173 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.270144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-tls\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:18.272485 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.272462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33113bd9-0d82-42ea-925a-1150385869eb-node-exporter-tls\") pod \"node-exporter-2cw7w\" (UID: \"33113bd9-0d82-42ea-925a-1150385869eb\") " pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:18.318799 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.318770    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-87nft"
Sep 15 09:08:18.338271 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.338226    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:18.429477 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.429398    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2cw7w"
Sep 15 09:08:18.442182 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:18.442148    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33113bd9_0d82_42ea_925a_1150385869eb.slice/crio-31331128d143e22814b61e6a6b8d890e028fb49b17bf8551de61158a3ac24484 WatchSource:0}: Error finding container 31331128d143e22814b61e6a6b8d890e028fb49b17bf8551de61158a3ac24484: Status 404 returned error can't find the container with id 31331128d143e22814b61e6a6b8d890e028fb49b17bf8551de61158a3ac24484
Sep 15 09:08:18.490163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.490134    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-jnbzd"]
Sep 15 09:08:18.494289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.494267    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:18.496530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.496513    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 09:08:18.496676 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.496658    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 09:08:18.496730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.496678    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vk5gj\""
Sep 15 09:08:18.502213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.502191    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-jnbzd"]
Sep 15 09:08:18.508779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.508753    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:18.517428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.517408    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cw7w" event={"ID":"33113bd9-0d82-42ea-925a-1150385869eb","Type":"ContainerStarted","Data":"31331128d143e22814b61e6a6b8d890e028fb49b17bf8551de61158a3ac24484"}
Sep 15 09:08:18.519126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.519105    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl" event={"ID":"f00af179-f92d-4591-a7fe-41397aa3c5b1","Type":"ContainerStarted","Data":"a2d5385eee1d037b699acb5fc6c05711ba8bf1f40d5ae256cdcaae3cb4ca7da6"}
Sep 15 09:08:18.573242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.573196    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6927\" (UniqueName: \"kubernetes.io/projected/d3226dfa-ddd2-45f6-96b9-b03a879ba935-kube-api-access-q6927\") pod \"downloads-76f99584f4-jnbzd\" (UID: \"d3226dfa-ddd2-45f6-96b9-b03a879ba935\") " pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:18.608007 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.607974    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:18.632142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.632114    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 09:08:18.635594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.635578    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.638980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.638959    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 09:08:18.639146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.638959    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 09:08:18.639146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.638992    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 09:08:18.639146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639093    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 09:08:18.639305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639262    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 09:08:18.639305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639261    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 09:08:18.639397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639295    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 09:08:18.639397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639310    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 09:08:18.639397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639277    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bvhtn\""
Sep 15 09:08:18.639841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.639797    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 09:08:18.653675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.653650    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 09:08:18.674769 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.674714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6927\" (UniqueName: \"kubernetes.io/projected/d3226dfa-ddd2-45f6-96b9-b03a879ba935-kube-api-access-q6927\") pod \"downloads-76f99584f4-jnbzd\" (UID: \"d3226dfa-ddd2-45f6-96b9-b03a879ba935\") " pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:18.682938 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.682874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6927\" (UniqueName: \"kubernetes.io/projected/d3226dfa-ddd2-45f6-96b9-b03a879ba935-kube-api-access-q6927\") pod \"downloads-76f99584f4-jnbzd\" (UID: \"d3226dfa-ddd2-45f6-96b9-b03a879ba935\") " pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:18.702098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.702075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:18.775759 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775720    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.775950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.775950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775790    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-web-config\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.775950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtpv\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-kube-api-access-wbtpv\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.775950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.775950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-out\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.775974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.776021    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.776063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.776087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.776115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.776132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.776311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.776172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.801971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.801936    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:18.803886 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.803869    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:18.876518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.876492    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.876677 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.876531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.876677 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.876554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.876677 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.876570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-web-config\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.876726    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtpv\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-kube-api-access-wbtpv\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877288    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-out\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877462    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.877576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.877558    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.878462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.878441    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.878616 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.878502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.878789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.878763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.879564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.879507    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.879657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.879590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-web-config\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.879768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.879743    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.880345 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.880318    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.880723 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.880694    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.880961 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.880944    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-out\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.881052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.880951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.881425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.881409    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.881718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.881697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.884410 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.884393    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtpv\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-kube-api-access-wbtpv\") pod \"alertmanager-main-0\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.949276 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.949203    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:08:18.978054 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:18.978024    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:19.040787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.040754    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:19.077733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.077686    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-jnbzd"]
Sep 15 09:08:19.080417 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:19.080389    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3226dfa_ddd2_45f6_96b9_b03a879ba935.slice/crio-a06c561d5c1bc1d0815d0a0d3fe7f7f2dcc944f8cae33f4d69c928525d0bbb5f WatchSource:0}: Error finding container a06c561d5c1bc1d0815d0a0d3fe7f7f2dcc944f8cae33f4d69c928525d0bbb5f: Status 404 returned error can't find the container with id a06c561d5c1bc1d0815d0a0d3fe7f7f2dcc944f8cae33f4d69c928525d0bbb5f
Sep 15 09:08:19.234449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.234372    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 09:08:19.238095 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:19.238055    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e27a3c8_1147_4472_84da_aa8aaf209db7.slice/crio-f5f4d36fc2fba6893281975f83737a655c57d343b970e09d5bde1c30e8ef0d4f WatchSource:0}: Error finding container f5f4d36fc2fba6893281975f83737a655c57d343b970e09d5bde1c30e8ef0d4f: Status 404 returned error can't find the container with id f5f4d36fc2fba6893281975f83737a655c57d343b970e09d5bde1c30e8ef0d4f
Sep 15 09:08:19.426964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.426904    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:19.492146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.492060    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:19.496926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.496899    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"]
Sep 15 09:08:19.500364 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.500341    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.503013 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.502989    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 09:08:19.503142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.503047    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 09:08:19.503142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.502989    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fd9m5c8tetifh\""
Sep 15 09:08:19.503324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.503302    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 09:08:19.503409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.503375    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 09:08:19.503409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.503380    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 09:08:19.503513 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.503380    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q5gms\""
Sep 15 09:08:19.513144 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.513119    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"]
Sep 15 09:08:19.524604 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.524563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"f5f4d36fc2fba6893281975f83737a655c57d343b970e09d5bde1c30e8ef0d4f"}
Sep 15 09:08:19.526340 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.526311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-jnbzd" event={"ID":"d3226dfa-ddd2-45f6-96b9-b03a879ba935","Type":"ContainerStarted","Data":"a06c561d5c1bc1d0815d0a0d3fe7f7f2dcc944f8cae33f4d69c928525d0bbb5f"}
Sep 15 09:08:19.584765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.584638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.584765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.584679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtds\" (UniqueName: \"kubernetes.io/projected/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-kube-api-access-mxtds\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.584765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.584715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-tls\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.585016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.584783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-metrics-client-ca\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.585016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.584862    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.585016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.584897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.585239 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.585025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.585239 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.585057    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-grpc-tls\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.601263 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.601232    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:19.685576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-metrics-client-ca\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.685708 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.685708 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.685841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.685906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-grpc-tls\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.686164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685960    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.686164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.685994    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtds\" (UniqueName: \"kubernetes.io/projected/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-kube-api-access-mxtds\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.686164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.686032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-tls\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.686395 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.686346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-metrics-client-ca\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.688974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.688944    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.689075 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.688968    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-grpc-tls\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.689358 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.689332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.689453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.689391    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.689534 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.689513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-tls\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.690938 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.690917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.694391 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.694367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtds\" (UniqueName: \"kubernetes.io/projected/ff455cd0-448b-435d-b3f8-cfe300a9d9ab-kube-api-access-mxtds\") pod \"thanos-querier-d8ddb5c99-wg29g\" (UID: \"ff455cd0-448b-435d-b3f8-cfe300a9d9ab\") " pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:19.701061 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.701034    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:19.812852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:19.812742    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:20.111145 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:20.111040    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"]
Sep 15 09:08:20.247796 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:20.247759    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff455cd0_448b_435d_b3f8_cfe300a9d9ab.slice/crio-b9a7e00f3c8a74fdeeb402da4e5d72dd913f9a5adab60bd0e0e54b4b49694473 WatchSource:0}: Error finding container b9a7e00f3c8a74fdeeb402da4e5d72dd913f9a5adab60bd0e0e54b4b49694473: Status 404 returned error can't find the container with id b9a7e00f3c8a74fdeeb402da4e5d72dd913f9a5adab60bd0e0e54b4b49694473
Sep 15 09:08:20.532628 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:20.532532    2571 generic.go:356] "Generic (PLEG): container finished" podID="33113bd9-0d82-42ea-925a-1150385869eb" containerID="2a92fdef7bd768da91153efc072043bb75d2058a8e238d941424bb8692fa32f2" exitCode=0
Sep 15 09:08:20.532785 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:20.532617    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cw7w" event={"ID":"33113bd9-0d82-42ea-925a-1150385869eb","Type":"ContainerDied","Data":"2a92fdef7bd768da91153efc072043bb75d2058a8e238d941424bb8692fa32f2"}
Sep 15 09:08:20.534123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:20.534099    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"b9a7e00f3c8a74fdeeb402da4e5d72dd913f9a5adab60bd0e0e54b4b49694473"}
Sep 15 09:08:21.152132 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.152091    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:21.253623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.253588    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:21.544445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.544281    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cw7w" event={"ID":"33113bd9-0d82-42ea-925a-1150385869eb","Type":"ContainerStarted","Data":"d0a2166a5e6bdfce98c0087cb33968c4f6721bdccdfea1ce3e154e869d515104"}
Sep 15 09:08:21.544445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.544354    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2cw7w" event={"ID":"33113bd9-0d82-42ea-925a-1150385869eb","Type":"ContainerStarted","Data":"8b3b21aa9d2ef286dda6bfddc75d537092fc4c5913f4546b46014570d0393cba"}
Sep 15 09:08:21.547374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.547343    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl" event={"ID":"f00af179-f92d-4591-a7fe-41397aa3c5b1","Type":"ContainerStarted","Data":"5403472ca468a2b49185667e9780f90e74599255173e27ab3d75eb8b2a175449"}
Sep 15 09:08:21.547511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.547405    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl" event={"ID":"f00af179-f92d-4591-a7fe-41397aa3c5b1","Type":"ContainerStarted","Data":"952a9170d996ab31258afe0b01ebe7f6633c79f3f8a028502fa581e445438dcd"}
Sep 15 09:08:21.547511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.547434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl" event={"ID":"f00af179-f92d-4591-a7fe-41397aa3c5b1","Type":"ContainerStarted","Data":"eeb8922524aa17a215e79586ee85b5e619204014480b261b162b730bbd386ea1"}
Sep 15 09:08:21.550821 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.550381    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e" exitCode=0
Sep 15 09:08:21.550821 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.550467    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"}
Sep 15 09:08:21.564338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.564280    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2cw7w" podStartSLOduration=3.429276917 podStartE2EDuration="4.564259728s" totalImagesPullingTime="1.063441731s" totalInitContainerRuntime="71.54108ms" isStatefulPod=true podCreationTimestamp="2026-09-15 09:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:21.562553211 +0000 UTC m=+94.021842752" watchObservedRunningTime="2026-09-15 09:08:21.564259728 +0000 UTC m=+94.023549272"
Sep 15 09:08:21.585093 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.584216    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fn6dl" podStartSLOduration=2.124619899 podStartE2EDuration="4.584196095s" totalImagesPullingTime="2.459576196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:21.584092834 +0000 UTC m=+94.043382375" watchObservedRunningTime="2026-09-15 09:08:21.584196095 +0000 UTC m=+94.043485630"
Sep 15 09:08:21.805164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.805074    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:21.817722 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.817689    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"]
Sep 15 09:08:21.821357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.821328    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:21.824146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.824123    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j5jfl\""
Sep 15 09:08:21.824146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.824125    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 09:08:21.824327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.824156    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 09:08:21.824327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.824195    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 09:08:21.824327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.824254    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 09:08:21.824628 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.824606    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-42d7l3mc759s4\""
Sep 15 09:08:21.830485 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:21.830456    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"]
Sep 15 09:08:22.006885 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.006839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a66d263-45db-41fb-8896-fc3dc7141efa-audit-log\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.007120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.006899    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmv8v\" (UniqueName: \"kubernetes.io/projected/9a66d263-45db-41fb-8896-fc3dc7141efa-kube-api-access-fmv8v\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.007120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.006945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a66d263-45db-41fb-8896-fc3dc7141efa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.007120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.006986    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a66d263-45db-41fb-8896-fc3dc7141efa-metrics-server-audit-profiles\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.007120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.007039    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-client-ca-bundle\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.007120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.007073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-secret-metrics-server-client-certs\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.007120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.007106    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-secret-metrics-server-tls\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.107756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.107710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-client-ca-bundle\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.107959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.107872    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-secret-metrics-server-client-certs\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.107959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.107920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-secret-metrics-server-tls\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.108083 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.108004    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a66d263-45db-41fb-8896-fc3dc7141efa-audit-log\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.108083 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.108036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmv8v\" (UniqueName: \"kubernetes.io/projected/9a66d263-45db-41fb-8896-fc3dc7141efa-kube-api-access-fmv8v\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.108162 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.108083    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a66d263-45db-41fb-8896-fc3dc7141efa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.108162 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.108117    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a66d263-45db-41fb-8896-fc3dc7141efa-metrics-server-audit-profiles\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.108761 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.108732    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9a66d263-45db-41fb-8896-fc3dc7141efa-audit-log\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.108986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.108962    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a66d263-45db-41fb-8896-fc3dc7141efa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.109363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.109270    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9a66d263-45db-41fb-8896-fc3dc7141efa-metrics-server-audit-profiles\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.112261 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.112207    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-client-ca-bundle\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.115042 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.115017    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-secret-metrics-server-client-certs\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.117453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.117426    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmv8v\" (UniqueName: \"kubernetes.io/projected/9a66d263-45db-41fb-8896-fc3dc7141efa-kube-api-access-fmv8v\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.117980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.117936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9a66d263-45db-41fb-8896-fc3dc7141efa-secret-metrics-server-tls\") pod \"metrics-server-57f6c54bd6-fhnvc\" (UID: \"9a66d263-45db-41fb-8896-fc3dc7141efa\") " pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.133391 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.133327    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:22.299208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.299164    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:22.868278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:22.868151    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"]
Sep 15 09:08:23.116672 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:23.116616    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a66d263_45db_41fb_8896_fc3dc7141efa.slice/crio-5b222e7b50f36d2392ca95d57be74090bcd7f8bfc51aeec30a09c01c484f83f6 WatchSource:0}: Error finding container 5b222e7b50f36d2392ca95d57be74090bcd7f8bfc51aeec30a09c01c484f83f6: Status 404 returned error can't find the container with id 5b222e7b50f36d2392ca95d57be74090bcd7f8bfc51aeec30a09c01c484f83f6
Sep 15 09:08:23.263488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.263456    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:23.320966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.320921    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:23.567321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.567282    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"}
Sep 15 09:08:23.567499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.567371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"}
Sep 15 09:08:23.567499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.567414    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"}
Sep 15 09:08:23.567499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.567456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"}
Sep 15 09:08:23.567499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.567495    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"}
Sep 15 09:08:23.571028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.570997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"35cbb1f55a42cbf4cb1fcddc6514b49c823861a76263b94a51c1f651cac2bddd"}
Sep 15 09:08:23.571150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.571074    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"6a4da741cd23d28b181ff6c8e2f2edb5d85196de743613082ac41c755c62ef33"}
Sep 15 09:08:23.571150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.571117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"a1b72a6cd12c9b60259d7c7058c64ff2415382b204fd70ddfd07caf269e13e01"}
Sep 15 09:08:23.572468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.572436    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc" event={"ID":"9a66d263-45db-41fb-8896-fc3dc7141efa","Type":"ContainerStarted","Data":"5b222e7b50f36d2392ca95d57be74090bcd7f8bfc51aeec30a09c01c484f83f6"}
Sep 15 09:08:23.690605 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.690521    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:08:23.694766 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.694738    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:23.695100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.695077    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.697869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.697819    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.698161    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g2pj2\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.698431    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.699038    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.699489    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.699674    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.700002    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.700160    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.700221    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3p0n8n7cms18o\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.700430    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.700644    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.700920    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.701256    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.701430    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 09:08:23.701953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.701656    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 09:08:23.702754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.702036    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 09:08:23.718267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.718234    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:08:23.826488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7v47\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-kube-api-access-s7v47\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826621    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826823    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-config\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.826915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826871    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.826982    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827142    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827168    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.827663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.827200    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928189    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928283    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928308    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928335    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928462    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7v47\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-kube-api-access-s7v47\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.928639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.929131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.929131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-config\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.929131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928709    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.929131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.928747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.929873 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.929849    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.932471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.932445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.933202 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.932754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.935673 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.935414    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.936126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.936096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.936819 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.936777    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.938747 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.938409    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.941387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.941329    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.941498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.941475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.941604 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.941380    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.941939 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.941917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7v47\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-kube-api-access-s7v47\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.942331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.942309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.943222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.943123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.943327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.943241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.945777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.945278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.947545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.947520    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-config\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.947928 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.947895    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.950406 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.950382    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.953555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.952709    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:23.954760 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:23.954539    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:24.014129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:24.013592    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:24.967986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:24.967940    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:08:24.968369 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:24.968304    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2286b8c_5686_4f9e_928f_1a963c30ef22.slice/crio-19a8ac7cebb6c437fa2f27922d82bee499cc9096b05262c0e7506ea04334a676 WatchSource:0}: Error finding container 19a8ac7cebb6c437fa2f27922d82bee499cc9096b05262c0e7506ea04334a676: Status 404 returned error can't find the container with id 19a8ac7cebb6c437fa2f27922d82bee499cc9096b05262c0e7506ea04334a676
Sep 15 09:08:25.585210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.585170    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc" event={"ID":"9a66d263-45db-41fb-8896-fc3dc7141efa","Type":"ContainerStarted","Data":"0b7079f547731a5a69d7022e454e7c162f640d24e45ddabb2cc022f813bc8d88"}
Sep 15 09:08:25.589309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.589225    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerStarted","Data":"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"}
Sep 15 09:08:25.590971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.590941    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5" exitCode=0
Sep 15 09:08:25.591115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.591025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"}
Sep 15 09:08:25.591115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.591092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"19a8ac7cebb6c437fa2f27922d82bee499cc9096b05262c0e7506ea04334a676"}
Sep 15 09:08:25.594678 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.594650    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"6eb12178e209b425efc3abddbfca8c3827ca1bffc4fea873f423d52cf5ab6882"}
Sep 15 09:08:25.594777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.594719    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"14c0882ad9af6c7cde266e140fa97e71bc5b59389446365e8a2ac17b08e43be6"}
Sep 15 09:08:25.594777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.594767    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:25.595035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.594808    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" event={"ID":"ff455cd0-448b-435d-b3f8-cfe300a9d9ab","Type":"ContainerStarted","Data":"b085cb7b8ceb8ecb22f46f95b5670b1a9bdc2d2477684bd019f2c2ec2cd12d45"}
Sep 15 09:08:25.603193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.603144    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc" podStartSLOduration=3.038582176 podStartE2EDuration="4.603126743s" totalImagesPullingTime="1.564544567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:25.602642968 +0000 UTC m=+98.061932520" watchObservedRunningTime="2026-09-15 09:08:25.603126743 +0000 UTC m=+98.062416283"
Sep 15 09:08:25.651351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.651286    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.489988018 podStartE2EDuration="7.651264334s" totalImagesPullingTime="4.138254714s" totalInitContainerRuntime="23.021602ms" isStatefulPod=false podCreationTimestamp="2026-09-15 09:08:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:25.650420562 +0000 UTC m=+98.109710103" watchObservedRunningTime="2026-09-15 09:08:25.651264334 +0000 UTC m=+98.110553876"
Sep 15 09:08:25.679719 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:25.677220    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g" podStartSLOduration=2.842037507 podStartE2EDuration="6.677202989s" totalImagesPullingTime="3.835165482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:08:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:25.675950447 +0000 UTC m=+98.135239985" watchObservedRunningTime="2026-09-15 09:08:25.677202989 +0000 UTC m=+98.136492530"
Sep 15 09:08:26.426279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:26.426243    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-785dcc4fcf-b6mzw"]
Sep 15 09:08:26.494556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:26.494515    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:26.595856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:26.595800    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:26.651491 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:26.651445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:27.216717 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:27.216680    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:28.660787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:28.660705    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:29.621303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:29.621271    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"}
Sep 15 09:08:29.621418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:29.621344    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"}
Sep 15 09:08:29.621418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:29.621373    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"}
Sep 15 09:08:30.337248 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:30.337212    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:30.437514 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:30.437471    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:30.630464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:30.630373    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"}
Sep 15 09:08:30.630464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:30.630456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"}
Sep 15 09:08:30.630690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:30.630499    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerStarted","Data":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"}
Sep 15 09:08:30.659652 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:30.659592    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.035455892 podStartE2EDuration="7.659570551s" totalImagesPullingTime="3.602280058s" totalInitContainerRuntime="21.834601ms" isStatefulPod=false podCreationTimestamp="2026-09-15 09:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:30.65667521 +0000 UTC m=+103.115964753" watchObservedRunningTime="2026-09-15 09:08:30.659570551 +0000 UTC m=+103.118860092"
Sep 15 09:08:31.442791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:31.442747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xd2s6"
Sep 15 09:08:31.607627 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:31.607560    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d8ddb5c99-wg29g"
Sep 15 09:08:33.009160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:33.009130    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:33.066561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:33.066526    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:34.020750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:34.020713    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:08:35.119432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:35.119389    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:35.219611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:35.219575    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:36.612034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:36.611997    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:36.711756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:36.711720    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:37.667188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:37.667148    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-jnbzd" event={"ID":"d3226dfa-ddd2-45f6-96b9-b03a879ba935","Type":"ContainerStarted","Data":"59600844a739f614a3652d679d1db5a44f7abdb818d19a6b3269336c08fb8e7f"}
Sep 15 09:08:37.667677 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:37.667309    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:37.685463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:37.685433    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-jnbzd"
Sep 15 09:08:37.686644 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:37.686594    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-jnbzd" podStartSLOduration=1.5545151320000001 podStartE2EDuration="19.686579133s" totalImagesPullingTime="18.132064001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:08:37.68596604 +0000 UTC m=+110.145255583" watchObservedRunningTime="2026-09-15 09:08:37.686579133 +0000 UTC m=+110.145868708"
Sep 15 09:08:39.412235 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:39.412195    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:39.466461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:39.466418    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:39.869069 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:39.869031    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:39.937731 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:39.937678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:41.209078 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:41.209044    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:41.261611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:41.261571    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:42.134203 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:42.134164    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:42.134398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:42.134239    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:08:42.216471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:42.216396    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:42.272170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:42.272138    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:46.384919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:46.384884    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:46.997307 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:46.997273    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:47.099457 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:47.099427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:50.727874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:50.727772    2571 generic.go:356] "Generic (PLEG): container finished" podID="31b4dd4b-c91c-4e10-87cf-7dd643e2edee" containerID="bab26ace3b6acaaa767ef53b9692ce480f44f70bee96a4b4e5a866ab57117896" exitCode=0
Sep 15 09:08:50.727874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:50.727846    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td" event={"ID":"31b4dd4b-c91c-4e10-87cf-7dd643e2edee","Type":"ContainerDied","Data":"bab26ace3b6acaaa767ef53b9692ce480f44f70bee96a4b4e5a866ab57117896"}
Sep 15 09:08:50.728324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:50.728184    2571 scope.go:117] "RemoveContainer" containerID="bab26ace3b6acaaa767ef53b9692ce480f44f70bee96a4b4e5a866ab57117896"
Sep 15 09:08:51.453956 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.453919    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" podUID="dd2aced5-6d03-4660-9c20-cf2248046c9c" containerName="registry" containerID="cri-o://63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53" gracePeriod=30
Sep 15 09:08:51.724212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.724144    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:08:51.732992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.732965    2571 generic.go:356] "Generic (PLEG): container finished" podID="dd2aced5-6d03-4660-9c20-cf2248046c9c" containerID="63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53" exitCode=0
Sep 15 09:08:51.733381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.733005    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw"
Sep 15 09:08:51.733381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.733046    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" event={"ID":"dd2aced5-6d03-4660-9c20-cf2248046c9c","Type":"ContainerDied","Data":"63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53"}
Sep 15 09:08:51.733381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.733091    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-785dcc4fcf-b6mzw" event={"ID":"dd2aced5-6d03-4660-9c20-cf2248046c9c","Type":"ContainerDied","Data":"4752c0ae3d4cc96bcf06fc7a801a68b3353ea0019304edfdf623d38ecfc59cd8"}
Sep 15 09:08:51.733381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.733117    2571 scope.go:117] "RemoveContainer" containerID="63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53"
Sep 15 09:08:51.734963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.734942    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-cj2td" event={"ID":"31b4dd4b-c91c-4e10-87cf-7dd643e2edee","Type":"ContainerStarted","Data":"ae90ef166db9ce927ee0db127bc75f0887695b627cc060100bf1bb3865026fff"}
Sep 15 09:08:51.741084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.741068    2571 scope.go:117] "RemoveContainer" containerID="63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53"
Sep 15 09:08:51.741349 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:08:51.741328    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53\": container with ID starting with 63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53 not found: ID does not exist" containerID="63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53"
Sep 15 09:08:51.741405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.741359    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53"} err="failed to get container status \"63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53\": rpc error: code = NotFound desc = could not find container \"63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53\": container with ID starting with 63bbd116fd67c7691bdcfaf387c153b8a7e6259979e2089dea67e34a093b5a53 not found: ID does not exist"
Sep 15 09:08:51.812133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-bound-sa-token\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.812304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812184    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd2aced5-6d03-4660-9c20-cf2248046c9c-ca-trust-extracted\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.812304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812235    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-installation-pull-secrets\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.812304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812263    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-image-registry-private-configuration\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.812304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812296    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xkjn\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-kube-api-access-4xkjn\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.812554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812351    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.812554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812381    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-certificates\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.813754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812675    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-trusted-ca\") pod \"dd2aced5-6d03-4660-9c20-cf2248046c9c\" (UID: \"dd2aced5-6d03-4660-9c20-cf2248046c9c\") "
Sep 15 09:08:51.813754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.812841    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:08:51.813754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.813185    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-certificates\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.813754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.813282    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:08:51.815332 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.815162    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:08:51.815332 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.815224    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:08:51.815332 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.815295    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:08:51.815584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.815553    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-kube-api-access-4xkjn" (OuterVolumeSpecName: "kube-api-access-4xkjn") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "kube-api-access-4xkjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:08:51.815978 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.815957    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:08:51.822602 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.822577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2aced5-6d03-4660-9c20-cf2248046c9c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dd2aced5-6d03-4660-9c20-cf2248046c9c" (UID: "dd2aced5-6d03-4660-9c20-cf2248046c9c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:08:51.913970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.913931    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-bound-sa-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.913970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.913964    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd2aced5-6d03-4660-9c20-cf2248046c9c-ca-trust-extracted\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.913970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.913974    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-installation-pull-secrets\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.914210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.913983    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd2aced5-6d03-4660-9c20-cf2248046c9c-image-registry-private-configuration\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.914210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.913993    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xkjn\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-kube-api-access-4xkjn\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.914210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.914003    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd2aced5-6d03-4660-9c20-cf2248046c9c-registry-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:51.914210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:51.914011    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd2aced5-6d03-4660-9c20-cf2248046c9c-trusted-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:08:52.058842 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:52.058807    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-785dcc4fcf-b6mzw"]
Sep 15 09:08:52.060681 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:52.060629    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-785dcc4fcf-b6mzw"]
Sep 15 09:08:54.054031 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:54.053988    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd2aced5-6d03-4660-9c20-cf2248046c9c" path="/var/lib/kubelet/pods/dd2aced5-6d03-4660-9c20-cf2248046c9c/volumes"
Sep 15 09:08:55.139082 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:55.139046    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:55.241186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:55.241147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:08:57.869397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:57.869352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:08:57.871667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:57.871639    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84527138-dafc-4435-b121-9215368bcb16-metrics-certs\") pod \"network-metrics-daemon-d8h8m\" (UID: \"84527138-dafc-4435-b121-9215368bcb16\") " pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:08:58.160489 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:58.160410    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fwjtr\""
Sep 15 09:08:58.168391 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:58.168369    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8h8m"
Sep 15 09:08:58.480375 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:58.480291    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8h8m"]
Sep 15 09:08:58.484308 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:08:58.484272    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84527138_dafc_4435_b121_9215368bcb16.slice/crio-740035d0594a9f96c85fefd5697114e9238a2bf04251d44fa9b6e06d6a1b4bd8 WatchSource:0}: Error finding container 740035d0594a9f96c85fefd5697114e9238a2bf04251d44fa9b6e06d6a1b4bd8: Status 404 returned error can't find the container with id 740035d0594a9f96c85fefd5697114e9238a2bf04251d44fa9b6e06d6a1b4bd8
Sep 15 09:08:58.764865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:08:58.764759    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8h8m" event={"ID":"84527138-dafc-4435-b121-9215368bcb16","Type":"ContainerStarted","Data":"740035d0594a9f96c85fefd5697114e9238a2bf04251d44fa9b6e06d6a1b4bd8"}
Sep 15 09:09:00.775567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:00.775533    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8h8m" event={"ID":"84527138-dafc-4435-b121-9215368bcb16","Type":"ContainerStarted","Data":"342b8e96ec76c805d215558152188b6fcf0d66a0cb4da38593baadb8c83dcd9a"}
Sep 15 09:09:00.776031 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:00.775605    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8h8m" event={"ID":"84527138-dafc-4435-b121-9215368bcb16","Type":"ContainerStarted","Data":"9672cbd4545e85303fb5758512f2bbeeeca9b72bb7416f5e5ef8a21232d48bea"}
Sep 15 09:09:00.796601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:00.796536    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d8h8m" podStartSLOduration=130.259912477 podStartE2EDuration="2m11.796511864s" totalImagesPullingTime="1.536599387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:09:00.794077837 +0000 UTC m=+133.253367381" watchObservedRunningTime="2026-09-15 09:09:00.796511864 +0000 UTC m=+133.255801405"
Sep 15 09:09:00.883031 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:00.882998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:00.939707 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:00.939675    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:02.140197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:02.140163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:09:02.144202 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:02.144176    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57f6c54bd6-fhnvc"
Sep 15 09:09:09.572140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:09.572105    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:09.672975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:09.672943    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:24.014218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:24.014167    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:24.030052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:24.030024    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:24.891789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:24.891761    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:33.635544 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:33.635485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:33.737040 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:33.737002    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:37.841382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841342    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 09:09:37.841784 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841622    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="alertmanager" containerID="cri-o://ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9" gracePeriod=120
Sep 15 09:09:37.841784 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841712    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-web" containerID="cri-o://13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b" gracePeriod=120
Sep 15 09:09:37.841784 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841717    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy" containerID="cri-o://46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d" gracePeriod=120
Sep 15 09:09:37.841784 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841771    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="prom-label-proxy" containerID="cri-o://9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4" gracePeriod=120
Sep 15 09:09:37.842004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841778    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-metric" containerID="cri-o://89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47" gracePeriod=120
Sep 15 09:09:37.842004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:37.841746    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="config-reloader" containerID="cri-o://ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f" gracePeriod=120
Sep 15 09:09:38.935575 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935549    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4" exitCode=0
Sep 15 09:09:38.935575 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935574    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d" exitCode=0
Sep 15 09:09:38.935912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935581    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f" exitCode=0
Sep 15 09:09:38.935912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935587    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9" exitCode=0
Sep 15 09:09:38.935912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"}
Sep 15 09:09:38.935912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935712    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"}
Sep 15 09:09:38.935912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935755    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"}
Sep 15 09:09:38.935912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:38.935781    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"}
Sep 15 09:09:39.085399 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.085371    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:09:39.247528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247438    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-web\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247491    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247511    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-main-tls\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247533    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbtpv\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-kube-api-access-wbtpv\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247563    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-metrics-client-ca\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247584    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-tls-assets\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-volume\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247638    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-trusted-ca-bundle\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247666    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247710    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-web-config\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247735    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-main-db\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.247803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247787    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-cluster-tls-config\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.248283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.247863    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-out\") pod \"9e27a3c8-1147-4472-84da-aa8aaf209db7\" (UID: \"9e27a3c8-1147-4472-84da-aa8aaf209db7\") "
Sep 15 09:09:39.248283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.248018    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:39.248283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.248149    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-metrics-client-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.248548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.248494    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:09:39.249022 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.248976    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:39.250478 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.250441    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.250563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.250494    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.250698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.250672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-kube-api-access-wbtpv" (OuterVolumeSpecName: "kube-api-access-wbtpv") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "kube-api-access-wbtpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:09:39.251064 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.251037    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-out" (OuterVolumeSpecName: "config-out") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:09:39.251225 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.251200    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-volume" (OuterVolumeSpecName: "config-volume") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.251395 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.251369    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:09:39.251869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.251847    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.252602 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.252577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.255733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.255609    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.262226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.262202    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-web-config" (OuterVolumeSpecName: "web-config") pod "9e27a3c8-1147-4472-84da-aa8aaf209db7" (UID: "9e27a3c8-1147-4472-84da-aa8aaf209db7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:39.348618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348580    2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-cluster-tls-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348610    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-out\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348623    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348634    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348645    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-main-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348654    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtpv\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-kube-api-access-wbtpv\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348662    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e27a3c8-1147-4472-84da-aa8aaf209db7-tls-assets\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348670    2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-config-volume\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348678    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348686    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348718    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e27a3c8-1147-4472-84da-aa8aaf209db7-web-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.348872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.348727    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e27a3c8-1147-4472-84da-aa8aaf209db7-alertmanager-main-db\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:39.943220 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943180    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47" exitCode=0
Sep 15 09:09:39.943220 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943206    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerID="13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b" exitCode=0
Sep 15 09:09:39.943745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"}
Sep 15 09:09:39.943745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943294    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 09:09:39.943745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"}
Sep 15 09:09:39.943745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943340    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e27a3c8-1147-4472-84da-aa8aaf209db7","Type":"ContainerDied","Data":"f5f4d36fc2fba6893281975f83737a655c57d343b970e09d5bde1c30e8ef0d4f"}
Sep 15 09:09:39.943745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.943378    2571 scope.go:117] "RemoveContainer" containerID="9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"
Sep 15 09:09:39.954542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.954346    2571 scope.go:117] "RemoveContainer" containerID="89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"
Sep 15 09:09:39.963018 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.962817    2571 scope.go:117] "RemoveContainer" containerID="46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"
Sep 15 09:09:39.970506 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.970478    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 09:09:39.970905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.970879    2571 scope.go:117] "RemoveContainer" containerID="13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"
Sep 15 09:09:39.974297 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.974276    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 09:09:39.978943 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.978923    2571 scope.go:117] "RemoveContainer" containerID="ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"
Sep 15 09:09:39.986019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.986001    2571 scope.go:117] "RemoveContainer" containerID="ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"
Sep 15 09:09:39.993377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:39.993350    2571 scope.go:117] "RemoveContainer" containerID="8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"
Sep 15 09:09:40.000920 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.000900    2571 scope.go:117] "RemoveContainer" containerID="9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"
Sep 15 09:09:40.001218 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.001198    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4\": container with ID starting with 9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4 not found: ID does not exist" containerID="9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"
Sep 15 09:09:40.001268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.001231    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"} err="failed to get container status \"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4\": rpc error: code = NotFound desc = could not find container \"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4\": container with ID starting with 9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4 not found: ID does not exist"
Sep 15 09:09:40.001268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.001251    2571 scope.go:117] "RemoveContainer" containerID="89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"
Sep 15 09:09:40.001529 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.001508    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47\": container with ID starting with 89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47 not found: ID does not exist" containerID="89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"
Sep 15 09:09:40.001588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.001540    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"} err="failed to get container status \"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47\": rpc error: code = NotFound desc = could not find container \"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47\": container with ID starting with 89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47 not found: ID does not exist"
Sep 15 09:09:40.001588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.001565    2571 scope.go:117] "RemoveContainer" containerID="46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"
Sep 15 09:09:40.001817 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.001800    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d\": container with ID starting with 46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d not found: ID does not exist" containerID="46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"
Sep 15 09:09:40.001874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.001822    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"} err="failed to get container status \"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d\": rpc error: code = NotFound desc = could not find container \"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d\": container with ID starting with 46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d not found: ID does not exist"
Sep 15 09:09:40.001874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.001862    2571 scope.go:117] "RemoveContainer" containerID="13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"
Sep 15 09:09:40.002118 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.002099    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b\": container with ID starting with 13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b not found: ID does not exist" containerID="13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"
Sep 15 09:09:40.002205 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002120    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"} err="failed to get container status \"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b\": rpc error: code = NotFound desc = could not find container \"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b\": container with ID starting with 13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b not found: ID does not exist"
Sep 15 09:09:40.002205 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002133    2571 scope.go:117] "RemoveContainer" containerID="ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"
Sep 15 09:09:40.002375 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.002360    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f\": container with ID starting with ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f not found: ID does not exist" containerID="ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"
Sep 15 09:09:40.002418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002377    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"} err="failed to get container status \"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f\": rpc error: code = NotFound desc = could not find container \"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f\": container with ID starting with ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f not found: ID does not exist"
Sep 15 09:09:40.002418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002390    2571 scope.go:117] "RemoveContainer" containerID="ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"
Sep 15 09:09:40.002633 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.002619    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9\": container with ID starting with ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9 not found: ID does not exist" containerID="ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"
Sep 15 09:09:40.002682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002635    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"} err="failed to get container status \"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9\": rpc error: code = NotFound desc = could not find container \"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9\": container with ID starting with ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9 not found: ID does not exist"
Sep 15 09:09:40.002682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002647    2571 scope.go:117] "RemoveContainer" containerID="8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"
Sep 15 09:09:40.002889 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:40.002874    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e\": container with ID starting with 8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e not found: ID does not exist" containerID="8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"
Sep 15 09:09:40.002923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002894    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"} err="failed to get container status \"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e\": rpc error: code = NotFound desc = could not find container \"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e\": container with ID starting with 8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e not found: ID does not exist"
Sep 15 09:09:40.002923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.002909    2571 scope.go:117] "RemoveContainer" containerID="9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"
Sep 15 09:09:40.003115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003092    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4"} err="failed to get container status \"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4\": rpc error: code = NotFound desc = could not find container \"9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4\": container with ID starting with 9eea64195b3e83f16ee700e78119f4d6bcc04285b5cd147f55af9fd4ada6a2a4 not found: ID does not exist"
Sep 15 09:09:40.003115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003109    2571 scope.go:117] "RemoveContainer" containerID="89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"
Sep 15 09:09:40.003302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003286    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47"} err="failed to get container status \"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47\": rpc error: code = NotFound desc = could not find container \"89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47\": container with ID starting with 89666fa1f468be68f9c11ca69ad3186f43fe7e766d722b954d028b92ec2dfd47 not found: ID does not exist"
Sep 15 09:09:40.003302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003301    2571 scope.go:117] "RemoveContainer" containerID="46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"
Sep 15 09:09:40.003518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003503    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d"} err="failed to get container status \"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d\": rpc error: code = NotFound desc = could not find container \"46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d\": container with ID starting with 46dd831383926ff0a243ff43d4db375ad9a116029d1bc17119dd7568d9daff6d not found: ID does not exist"
Sep 15 09:09:40.003564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003518    2571 scope.go:117] "RemoveContainer" containerID="13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"
Sep 15 09:09:40.003745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003730    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b"} err="failed to get container status \"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b\": rpc error: code = NotFound desc = could not find container \"13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b\": container with ID starting with 13a7b0a22b4d35d8ba31510b1c183f79dc4da92b5728ae49b6339c19a5e6864b not found: ID does not exist"
Sep 15 09:09:40.003783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003746    2571 scope.go:117] "RemoveContainer" containerID="ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"
Sep 15 09:09:40.003946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003930    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f"} err="failed to get container status \"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f\": rpc error: code = NotFound desc = could not find container \"ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f\": container with ID starting with ac989cb04e125f2786df6f916dd528eea50d61c5108971cf285b13498f51383f not found: ID does not exist"
Sep 15 09:09:40.003946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.003945    2571 scope.go:117] "RemoveContainer" containerID="ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"
Sep 15 09:09:40.004121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.004104    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9"} err="failed to get container status \"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9\": rpc error: code = NotFound desc = could not find container \"ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9\": container with ID starting with ae268b205f09215963d5b6b40a9a7581681e96189f5ccc899d705d97f3dcc1b9 not found: ID does not exist"
Sep 15 09:09:40.004165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.004121    2571 scope.go:117] "RemoveContainer" containerID="8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"
Sep 15 09:09:40.004344 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.004329    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e"} err="failed to get container status \"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e\": rpc error: code = NotFound desc = could not find container \"8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e\": container with ID starting with 8a0af95dacc22fded7778478948242504f53a487a5acd368a5bae8e345bdf18e not found: ID does not exist"
Sep 15 09:09:40.053865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:40.053813    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" path="/var/lib/kubelet/pods/9e27a3c8-1147-4472-84da-aa8aaf209db7/volumes"
Sep 15 09:09:42.105443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105411    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:09:42.105900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105703    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="prometheus" containerID="cri-o://fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f" gracePeriod=600
Sep 15 09:09:42.105900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105756    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-thanos" containerID="cri-o://947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e" gracePeriod=600
Sep 15 09:09:42.105900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105761    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="thanos-sidecar" containerID="cri-o://518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066" gracePeriod=600
Sep 15 09:09:42.105900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105775    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="config-reloader" containerID="cri-o://897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916" gracePeriod=600
Sep 15 09:09:42.105900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105706    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy" containerID="cri-o://cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794" gracePeriod=600
Sep 15 09:09:42.105900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.105775    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-web" containerID="cri-o://a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6" gracePeriod=600
Sep 15 09:09:42.368674 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.368648    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:42.477049 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477010    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-web-config\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477049 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477049    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-0\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477071    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477090    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-2\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477110    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-thanos-prometheus-http-client-file\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477169    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-metrics-client-ca\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477193    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-config-out\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477232    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-metrics-client-certs\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477257    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-grpc-tls\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477298    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-trusted-ca-bundle\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477330    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477364    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-tls-assets\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477384    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477410    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-kubelet-serving-ca-bundle\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477478    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-tls\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477512    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7v47\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-kube-api-access-s7v47\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477552    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-serving-certs-ca-bundle\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477579    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-db\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.477622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477608    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-config\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477632    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-kube-rbac-proxy\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477658    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-1\") pod \"d2286b8c-5686-4f9e-928f-1a963c30ef22\" (UID: \"d2286b8c-5686-4f9e-928f-1a963c30ef22\") "
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477796    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.477813    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.478031    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.478056    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.478138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.478071    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.479795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.478759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.479795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.479071    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.479795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.479416    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.480163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.480142    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:09:42.480583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.480558    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.480679 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.480631    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:09:42.481225 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.481179    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.481323 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.481234    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.481713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.481672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.482018 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.481989    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.482393 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.482341    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.482487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.482467    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:09:42.482604 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.482582    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.482673 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.482626    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-config-out" (OuterVolumeSpecName: "config-out") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:09:42.482673 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.482640    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-config" (OuterVolumeSpecName: "config") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.482780 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.482695    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-kube-api-access-s7v47" (OuterVolumeSpecName: "kube-api-access-s7v47") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "kube-api-access-s7v47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:09:42.490244 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.490221    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-web-config" (OuterVolumeSpecName: "web-config") pod "d2286b8c-5686-4f9e-928f-1a963c30ef22" (UID: "d2286b8c-5686-4f9e-928f-1a963c30ef22"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:09:42.579304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579270    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579301    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7v47\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-kube-api-access-s7v47\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579312    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579322    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-db\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579332    2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579340    2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-kube-rbac-proxy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579349    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579357    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-web-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579366    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579375    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579383    2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579392    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2286b8c-5686-4f9e-928f-1a963c30ef22-configmap-metrics-client-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579401    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2286b8c-5686-4f9e-928f-1a963c30ef22-config-out\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579409    2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-metrics-client-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579417    2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-grpc-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579425    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2286b8c-5686-4f9e-928f-1a963c30ef22-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.579516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.579432    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2286b8c-5686-4f9e-928f-1a963c30ef22-tls-assets\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:09:42.958810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958773    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e" exitCode=0
Sep 15 09:09:42.958810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958806    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794" exitCode=0
Sep 15 09:09:42.958810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958815    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6" exitCode=0
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958839    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066" exitCode=0
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958848    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916" exitCode=0
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958857    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f" exitCode=0
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958858    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"}
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958888    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958930    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"}
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"}
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.958985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"}
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.959001    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.959009    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"}
Sep 15 09:09:42.959046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.959034    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"}
Sep 15 09:09:42.959448 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.959058    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2286b8c-5686-4f9e-928f-1a963c30ef22","Type":"ContainerDied","Data":"19a8ac7cebb6c437fa2f27922d82bee499cc9096b05262c0e7506ea04334a676"}
Sep 15 09:09:42.967420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.967332    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:42.974579 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.974557    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:42.981740 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.981707    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:42.984222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.984198    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:09:42.989679 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.989653    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:09:42.990771 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.990749    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:42.997710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:42.997692    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.005489 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.005438    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.012470 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.012449    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.012777 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.012756    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.012854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.012786    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"} err="failed to get container status \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist"
Sep 15 09:09:43.012854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.012808    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.013091 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.013068    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.013136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.013101    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"} err="failed to get container status \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist"
Sep 15 09:09:43.013136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.013127    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.013618 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.013481    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.013618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.013514    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"} err="failed to get container status \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist"
Sep 15 09:09:43.013618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.013535    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.014316 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.014196    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.014316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.014242    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"} err="failed to get container status \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist"
Sep 15 09:09:43.014316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.014260    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.014579 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.014554    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.014658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.014585    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"} err="failed to get container status \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist"
Sep 15 09:09:43.014658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.014605    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.014920 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.014896    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.014980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.014921    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"} err="failed to get container status \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist"
Sep 15 09:09:43.014980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.014935    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.015198 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:09:43.015178    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.015257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015202    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"} err="failed to get container status \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist"
Sep 15 09:09:43.015257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015217    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.015450 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015433    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"} err="failed to get container status \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist"
Sep 15 09:09:43.015501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015449    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.015571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015551    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:09:43.015703 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015685    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"} err="failed to get container status \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist"
Sep 15 09:09:43.015763 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015703    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.015916 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015902    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2aced5-6d03-4660-9c20-cf2248046c9c" containerName="registry"
Sep 15 09:09:43.015972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015922    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2aced5-6d03-4660-9c20-cf2248046c9c" containerName="registry"
Sep 15 09:09:43.015972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015937    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-thanos"
Sep 15 09:09:43.015972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015946    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-thanos"
Sep 15 09:09:43.015972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015960    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"} err="failed to get container status \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015978    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.015967    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="init-config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016025    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="init-config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016035    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-web"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016041    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-web"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016052    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016057    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016065    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-metric"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016070    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-metric"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016080    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="init-config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016089    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="init-config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016099    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016106    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016118    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="config-reloader"
Sep 15 09:09:43.016121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016124    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="config-reloader"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016134    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="thanos-sidecar"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016139    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="thanos-sidecar"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016149    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="alertmanager"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016157    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="alertmanager"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016169    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="prom-label-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016175    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="prom-label-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016183    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="prometheus"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016188    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="prometheus"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016197    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-web"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016203    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-web"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016208    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016213    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016219    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"} err="failed to get container status \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016234    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016277    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="prometheus"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016286    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2aced5-6d03-4660-9c20-cf2248046c9c" containerName="registry"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016292    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016297    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-metric"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016303    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-web"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016307    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="config-reloader"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016313    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="kube-rbac-proxy-web"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016322    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="config-reloader"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016329    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="thanos-sidecar"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016338    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="alertmanager"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016346    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy-thanos"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016356    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" containerName="kube-rbac-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016365    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e27a3c8-1147-4472-84da-aa8aaf209db7" containerName="prom-label-proxy"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016473    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"} err="failed to get container status \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist"
Sep 15 09:09:43.016657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016496    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016769    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"} err="failed to get container status \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.016801    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017067    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"} err="failed to get container status \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017086    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017330    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"} err="failed to get container status \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017350    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017556    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"} err="failed to get container status \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist"
Sep 15 09:09:43.017713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017572    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.018023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017800    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"} err="failed to get container status \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist"
Sep 15 09:09:43.018023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.017820    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.018023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018005    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"} err="failed to get container status \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist"
Sep 15 09:09:43.018023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018017    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.018283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018262    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"} err="failed to get container status \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist"
Sep 15 09:09:43.018340 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018283    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.018511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018494    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"} err="failed to get container status \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist"
Sep 15 09:09:43.018566 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018511    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.018718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018702    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"} err="failed to get container status \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist"
Sep 15 09:09:43.018764 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018717    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.018940 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018922    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"} err="failed to get container status \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist"
Sep 15 09:09:43.019000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.018941    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.019186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019159    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"} err="failed to get container status \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist"
Sep 15 09:09:43.019186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019178    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.019394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019377    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"} err="failed to get container status \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist"
Sep 15 09:09:43.019443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019394    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.019582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019567    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"} err="failed to get container status \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist"
Sep 15 09:09:43.019619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019582    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.019812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019789    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"} err="failed to get container status \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist"
Sep 15 09:09:43.019812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.019811    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.020040 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020023    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"} err="failed to get container status \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist"
Sep 15 09:09:43.020096 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020040    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.020244 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020228    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"} err="failed to get container status \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist"
Sep 15 09:09:43.020289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020244    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.020423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020402    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.020464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020444    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"} err="failed to get container status \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist"
Sep 15 09:09:43.020464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020459    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.020794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020669    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"} err="failed to get container status \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist"
Sep 15 09:09:43.020794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020696    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.021016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.020993    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"} err="failed to get container status \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist"
Sep 15 09:09:43.021079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021019    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.021310 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021288    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"} err="failed to get container status \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist"
Sep 15 09:09:43.021364 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021311    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.021574 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021556    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"} err="failed to get container status \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist"
Sep 15 09:09:43.021614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021575    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.021803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021784    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"} err="failed to get container status \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist"
Sep 15 09:09:43.021909 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.021804    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.022115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022098    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"} err="failed to get container status \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist"
Sep 15 09:09:43.022161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022116    2571 scope.go:117] "RemoveContainer" containerID="947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"
Sep 15 09:09:43.022387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022369    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e"} err="failed to get container status \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": rpc error: code = NotFound desc = could not find container \"947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e\": container with ID starting with 947d2032baecf7fcfa92e64f4f5180f62145c4648fbfe19e726c7ceab4256d4e not found: ID does not exist"
Sep 15 09:09:43.022445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022388    2571 scope.go:117] "RemoveContainer" containerID="cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"
Sep 15 09:09:43.022619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022602    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794"} err="failed to get container status \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": rpc error: code = NotFound desc = could not find container \"cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794\": container with ID starting with cf7f95d8d5f1eab3237056a309b3c187adaab7e47e3e25976385630d0a124794 not found: ID does not exist"
Sep 15 09:09:43.022682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022619    2571 scope.go:117] "RemoveContainer" containerID="a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"
Sep 15 09:09:43.022875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022853    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6"} err="failed to get container status \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": rpc error: code = NotFound desc = could not find container \"a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6\": container with ID starting with a1afd1f7cd9b065d2a225d9cc716b7c43ea84cfd8b3b9d53adf14d5bf147d3f6 not found: ID does not exist"
Sep 15 09:09:43.022923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.022876    2571 scope.go:117] "RemoveContainer" containerID="518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"
Sep 15 09:09:43.023100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023081    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066"} err="failed to get container status \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": rpc error: code = NotFound desc = could not find container \"518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066\": container with ID starting with 518d3aa13b797f150fa02fd422344dcbed5ee0a0539aecb156472f4d2f920066 not found: ID does not exist"
Sep 15 09:09:43.023167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023099    2571 scope.go:117] "RemoveContainer" containerID="897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"
Sep 15 09:09:43.023311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023296    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916"} err="failed to get container status \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": rpc error: code = NotFound desc = could not find container \"897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916\": container with ID starting with 897687d47b8df952e90e36e971e34be453538a3380ed207e9b88ea3b380c3916 not found: ID does not exist"
Sep 15 09:09:43.023311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023310    2571 scope.go:117] "RemoveContainer" containerID="fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"
Sep 15 09:09:43.023541 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023524    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f"} err="failed to get container status \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": rpc error: code = NotFound desc = could not find container \"fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f\": container with ID starting with fc80ae48ac5127b5498637f3adec970b4b37f6496cb84a75c0042579f249236f not found: ID does not exist"
Sep 15 09:09:43.023541 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023540    2571 scope.go:117] "RemoveContainer" containerID="81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"
Sep 15 09:09:43.023760 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.023744    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5"} err="failed to get container status \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": rpc error: code = NotFound desc = could not find container \"81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5\": container with ID starting with 81c823762050abf8a8234c07a52656a95fa0ab51e9d87fb23277fc98dc0c10e5 not found: ID does not exist"
Sep 15 09:09:43.026064 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026041    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 09:09:43.026160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026041    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 09:09:43.026436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 09:09:43.026436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026425    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 09:09:43.026563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026441    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 09:09:43.026563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026469    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 09:09:43.026563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026429    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3p0n8n7cms18o\""
Sep 15 09:09:43.026705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026580    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 09:09:43.026790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.026775    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g2pj2\""
Sep 15 09:09:43.027106 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.027087    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 09:09:43.027187 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.027171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 09:09:43.031081 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.031056    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 09:09:43.031172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.031100    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 09:09:43.031172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.031120    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 09:09:43.041560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.041532    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 09:09:43.043561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.043544    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 09:09:43.050989 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.050964    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:09:43.184105 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3842aabf-ddf6-4aba-ac93-6c3623eb6544-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184105 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184131    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184266    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3842aabf-ddf6-4aba-ac93-6c3623eb6544-config-out\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-web-config\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184398    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184518    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqdc\" (UniqueName: \"kubernetes.io/projected/3842aabf-ddf6-4aba-ac93-6c3623eb6544-kube-api-access-6mqdc\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184617    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184668    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-config\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.184851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.184683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285332    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqdc\" (UniqueName: \"kubernetes.io/projected/3842aabf-ddf6-4aba-ac93-6c3623eb6544-kube-api-access-6mqdc\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285723 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285603    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285723 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285631    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285723 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-config\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285723 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3842aabf-ddf6-4aba-ac93-6c3623eb6544-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3842aabf-ddf6-4aba-ac93-6c3623eb6544-config-out\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285923    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285950    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.285964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.286401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.285978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-web-config\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.286401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.286022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.286401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.286099    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.286401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.286309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.286598 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.286500    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.288975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.288640    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3842aabf-ddf6-4aba-ac93-6c3623eb6544-config-out\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.288975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.288673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.288975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.288700    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.288975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.288809    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-web-config\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.289246 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.289034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.289309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.289248    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.289309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.289283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.290122 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.289460    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.290122 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.290047    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.290122 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.290075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.290656 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.290622    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3842aabf-ddf6-4aba-ac93-6c3623eb6544-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.290941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.290915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3842aabf-ddf6-4aba-ac93-6c3623eb6544-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.291218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.291200    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.291552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.291533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-config\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.291613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.291589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3842aabf-ddf6-4aba-ac93-6c3623eb6544-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.293953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.293931    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqdc\" (UniqueName: \"kubernetes.io/projected/3842aabf-ddf6-4aba-ac93-6c3623eb6544-kube-api-access-6mqdc\") pod \"prometheus-k8s-0\" (UID: \"3842aabf-ddf6-4aba-ac93-6c3623eb6544\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.330714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.330671    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:43.611654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.611629    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 09:09:43.613304 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:09:43.613261    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3842aabf_ddf6_4aba_ac93_6c3623eb6544.slice/crio-80a9958a4f1415dfc971bbadcc6cff3e213fc52ce4d0dc2bffba974973443b39 WatchSource:0}: Error finding container 80a9958a4f1415dfc971bbadcc6cff3e213fc52ce4d0dc2bffba974973443b39: Status 404 returned error can't find the container with id 80a9958a4f1415dfc971bbadcc6cff3e213fc52ce4d0dc2bffba974973443b39
Sep 15 09:09:43.965216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.965119    2571 generic.go:356] "Generic (PLEG): container finished" podID="3842aabf-ddf6-4aba-ac93-6c3623eb6544" containerID="e5947b9f8f7f01217f5c64260647befbe074d03bc297f5a22696a18a87835fc7" exitCode=0
Sep 15 09:09:43.965216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.965160    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerDied","Data":"e5947b9f8f7f01217f5c64260647befbe074d03bc297f5a22696a18a87835fc7"}
Sep 15 09:09:43.965216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:43.965210    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"80a9958a4f1415dfc971bbadcc6cff3e213fc52ce4d0dc2bffba974973443b39"}
Sep 15 09:09:44.054771 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.054737    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2286b8c-5686-4f9e-928f-1a963c30ef22" path="/var/lib/kubelet/pods/d2286b8c-5686-4f9e-928f-1a963c30ef22/volumes"
Sep 15 09:09:44.972912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.972874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"34e95ad08489ba1f7981ad0ecc4fb2f26790696ad6571941592386c31471d60f"}
Sep 15 09:09:44.973287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.972941    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"2a668cb0b177a7424f6b5f4c0d4e0ec302cf13902d61d4293781878310eb1762"}
Sep 15 09:09:44.973287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.972970    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"b29bb4e12458ac339f283fae78a40fbd5ccb2bab637be6c5c84fa1819013cded"}
Sep 15 09:09:44.973287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.973000    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"b32a57bc0e6e6860f28c5b0bdce8f4c41634f48121e0d405af294bf07bf524ad"}
Sep 15 09:09:44.973287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.973037    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"99b9af87e27000fbb3d8d12bfe81f653c10566b1c3c2d98a189b1ddaca9b019d"}
Sep 15 09:09:44.973287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:44.973073    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3842aabf-ddf6-4aba-ac93-6c3623eb6544","Type":"ContainerStarted","Data":"ae0a7bef67f7f3cca9c81c0ebe3562820a2289b964363a6389e90ba932588cea"}
Sep 15 09:09:45.005200 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:45.005148    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.981432345 podStartE2EDuration="3.00513308s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.700735ms" isStatefulPod=false podCreationTimestamp="2026-09-15 09:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:09:45.003366275 +0000 UTC m=+177.462655817" watchObservedRunningTime="2026-09-15 09:09:45.00513308 +0000 UTC m=+177.464422620"
Sep 15 09:09:45.533007 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:45.532968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:45.631874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:45.631840    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:48.331539 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:48.331458    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:09:54.578069 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:54.578036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:54.764965 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:54.764929    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:54.821159 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:54.821126    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:56.469595 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:56.469560    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:09:56.664853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:09:56.664801    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:07.061545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:07.061503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:07.123413 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:07.123380    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:07.174127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:07.174096    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:18.356117 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:18.356081    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:18.407636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:18.407602    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:32.647486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:32.647455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:43.330972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:43.330932    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:10:43.346324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:43.346293    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:10:44.221006 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:44.220978    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 09:10:47.371620 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:47.371582    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:10:47.463608 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:10:47.463574    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:10.231291 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:10.231210    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:14.110185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:14.110147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:14.907588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:14.907552    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:17.693188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:17.693123    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:17.794121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:17.794089    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:19.625887 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:19.625852    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:19.738962 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:19.738927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:19.885886 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:19.885774    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:21.987083 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:21.987052    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:22.411453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:22.411419    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:23.387607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.387571    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"]
Sep 15 09:11:23.390993 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.390971    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.399090 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.393759    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9bw7z\""
Sep 15 09:11:23.399090 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.393789    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 09:11:23.399090 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.394146    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 09:11:23.401199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.401171    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"]
Sep 15 09:11:23.515095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.515052    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6c6r\" (UniqueName: \"kubernetes.io/projected/ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb-kube-api-access-d6c6r\") pod \"cert-manager-cainjector-559cdd8c87-25nnt\" (UID: \"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.515294 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.515110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-25nnt\" (UID: \"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.616395 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.616351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6c6r\" (UniqueName: \"kubernetes.io/projected/ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb-kube-api-access-d6c6r\") pod \"cert-manager-cainjector-559cdd8c87-25nnt\" (UID: \"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.616586 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.616427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-25nnt\" (UID: \"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.624460 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.624411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-25nnt\" (UID: \"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.624710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.624688    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6c6r\" (UniqueName: \"kubernetes.io/projected/ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb-kube-api-access-d6c6r\") pod \"cert-manager-cainjector-559cdd8c87-25nnt\" (UID: \"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.721856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.721729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"
Sep 15 09:11:23.989641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:23.989568    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-25nnt"]
Sep 15 09:11:23.993786 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:11:23.993757    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac8fdfc5_1af3_4c4c_9e8d_63b0bee01cdb.slice/crio-5f030de329e959583ff45bf6c60f84106d83fba00db74a77b1b5e737cae279e8 WatchSource:0}: Error finding container 5f030de329e959583ff45bf6c60f84106d83fba00db74a77b1b5e737cae279e8: Status 404 returned error can't find the container with id 5f030de329e959583ff45bf6c60f84106d83fba00db74a77b1b5e737cae279e8
Sep 15 09:11:24.356558 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:24.356522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt" event={"ID":"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb","Type":"ContainerStarted","Data":"5f030de329e959583ff45bf6c60f84106d83fba00db74a77b1b5e737cae279e8"}
Sep 15 09:11:24.479963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:24.479927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:24.539273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:24.539236    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:27.370930 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:27.370894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt" event={"ID":"ac8fdfc5-1af3-4c4c-9e8d-63b0bee01cdb","Type":"ContainerStarted","Data":"9c0be2d49a806faab452e7d783fed1645476e54f7e6f3c1cec3232fd1645d67f"}
Sep 15 09:11:27.387928 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:27.387882    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25nnt" podStartSLOduration=1.4100757 podStartE2EDuration="4.387863069s" totalImagesPullingTime="2.977787369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:11:27.386934023 +0000 UTC m=+279.846223567" watchObservedRunningTime="2026-09-15 09:11:27.387863069 +0000 UTC m=+279.847152611"
Sep 15 09:11:27.758517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:27.758431    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:27.847988 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:27.847953    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:29.210418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:29.210385    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:30.899464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:30.899428    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:31.006066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:31.006029    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:37.203302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:37.203262    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:37.303626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:37.303593    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:38.868924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:38.868883    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:47.938408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:47.938372    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:11:47.940053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:47.940029    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:11:47.954896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:47.954865    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 09:11:51.106848 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.106801    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:51.167499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.167458    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"]
Sep 15 09:11:51.170811 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.170794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.173619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.173596    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 09:11:51.174617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.174601    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 09:11:51.174815 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.174801    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8vtc5\""
Sep 15 09:11:51.174904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.174848    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 09:11:51.174904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.174870    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 09:11:51.174904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.174882    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 09:11:51.180621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.180595    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"]
Sep 15 09:11:51.188012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.187987    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:51.244850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.244797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/505e8ef8-30b0-462f-ad41-f747542f37ac-cert\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.245029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.244873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/505e8ef8-30b0-462f-ad41-f747542f37ac-metrics-cert\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.245029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.244925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/505e8ef8-30b0-462f-ad41-f747542f37ac-manager-config\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.245029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.244999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltm2\" (UniqueName: \"kubernetes.io/projected/505e8ef8-30b0-462f-ad41-f747542f37ac-kube-api-access-wltm2\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.346194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.346144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/505e8ef8-30b0-462f-ad41-f747542f37ac-manager-config\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.346194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.346206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltm2\" (UniqueName: \"kubernetes.io/projected/505e8ef8-30b0-462f-ad41-f747542f37ac-kube-api-access-wltm2\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.346422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.346242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/505e8ef8-30b0-462f-ad41-f747542f37ac-cert\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.346422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.346277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/505e8ef8-30b0-462f-ad41-f747542f37ac-metrics-cert\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.346801 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.346771    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/505e8ef8-30b0-462f-ad41-f747542f37ac-manager-config\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.348798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.348775    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/505e8ef8-30b0-462f-ad41-f747542f37ac-metrics-cert\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.348915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.348803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/505e8ef8-30b0-462f-ad41-f747542f37ac-cert\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.354967 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.354936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltm2\" (UniqueName: \"kubernetes.io/projected/505e8ef8-30b0-462f-ad41-f747542f37ac-kube-api-access-wltm2\") pod \"lws-controller-manager-5f667c6f8-66z7p\" (UID: \"505e8ef8-30b0-462f-ad41-f747542f37ac\") " pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.481848 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.481738    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:51.765613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.765529    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"]
Sep 15 09:11:51.768362 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:11:51.768336    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505e8ef8_30b0_462f_ad41_f747542f37ac.slice/crio-3a7572083344c5a5c527088f0d9283c6b15c0c0f3bc8b5e336fb30688ca3428b WatchSource:0}: Error finding container 3a7572083344c5a5c527088f0d9283c6b15c0c0f3bc8b5e336fb30688ca3428b: Status 404 returned error can't find the container with id 3a7572083344c5a5c527088f0d9283c6b15c0c0f3bc8b5e336fb30688ca3428b
Sep 15 09:11:51.770417 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:51.770396    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:11:52.475116 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:52.475065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p" event={"ID":"505e8ef8-30b0-462f-ad41-f747542f37ac","Type":"ContainerStarted","Data":"3a7572083344c5a5c527088f0d9283c6b15c0c0f3bc8b5e336fb30688ca3428b"}
Sep 15 09:11:54.485381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:54.485289    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p" event={"ID":"505e8ef8-30b0-462f-ad41-f747542f37ac","Type":"ContainerStarted","Data":"5e62ee832e75ce93ff9f4483fb8d6d7b4fd005b4d3ad8f2aab2f6561d0ab141f"}
Sep 15 09:11:54.485801 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:54.485427    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:11:54.502548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:54.502495    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p" podStartSLOduration=1.125404934 podStartE2EDuration="3.502479459s" totalImagesPullingTime="2.377074525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:11:54.500960518 +0000 UTC m=+306.960250058" watchObservedRunningTime="2026-09-15 09:11:54.502479459 +0000 UTC m=+306.961769001"
Sep 15 09:11:57.245488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:57.245451    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:11:58.272473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.272435    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"]
Sep 15 09:11:58.275979 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.275961    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.278390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.278369    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9l42l\""
Sep 15 09:11:58.278538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.278370    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 09:11:58.278619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.278545    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 09:11:58.278619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.278548    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 09:11:58.278798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.278773    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 09:11:58.289369 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.289346    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"]
Sep 15 09:11:58.403029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.402992    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3b9355e-a720-4212-881d-58e0b6de970c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.403197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.403042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3b9355e-a720-4212-881d-58e0b6de970c-tmp\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.403197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.403106    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3b9355e-a720-4212-881d-58e0b6de970c-manifests\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.403197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.403135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3b9355e-a720-4212-881d-58e0b6de970c-webhook-cert\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.403197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.403159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frsg\" (UniqueName: \"kubernetes.io/projected/e3b9355e-a720-4212-881d-58e0b6de970c-kube-api-access-8frsg\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.504242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.504206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3b9355e-a720-4212-881d-58e0b6de970c-manifests\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.504242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.504250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3b9355e-a720-4212-881d-58e0b6de970c-webhook-cert\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.504452 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.504271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frsg\" (UniqueName: \"kubernetes.io/projected/e3b9355e-a720-4212-881d-58e0b6de970c-kube-api-access-8frsg\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.504452 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.504318    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3b9355e-a720-4212-881d-58e0b6de970c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.504452 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.504345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3b9355e-a720-4212-881d-58e0b6de970c-tmp\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.504740 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.504714    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e3b9355e-a720-4212-881d-58e0b6de970c-manifests\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.506680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.506647    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3b9355e-a720-4212-881d-58e0b6de970c-tmp\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.506813 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.506767    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3b9355e-a720-4212-881d-58e0b6de970c-webhook-cert\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.506813 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.506774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e3b9355e-a720-4212-881d-58e0b6de970c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.513253 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.513223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frsg\" (UniqueName: \"kubernetes.io/projected/e3b9355e-a720-4212-881d-58e0b6de970c-kube-api-access-8frsg\") pod \"opendatahub-operator-controller-manager-77dcfd7f98-6jqhk\" (UID: \"e3b9355e-a720-4212-881d-58e0b6de970c\") " pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.586588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.586556    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:11:58.891016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:58.890939    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"]
Sep 15 09:11:58.895464 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:11:58.895433    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b9355e_a720_4212_881d_58e0b6de970c.slice/crio-5da49ccb06d02628714425d8c9e17c9d4918c4417ab07de44427438749705854 WatchSource:0}: Error finding container 5da49ccb06d02628714425d8c9e17c9d4918c4417ab07de44427438749705854: Status 404 returned error can't find the container with id 5da49ccb06d02628714425d8c9e17c9d4918c4417ab07de44427438749705854
Sep 15 09:11:59.506000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:59.505963    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk" event={"ID":"e3b9355e-a720-4212-881d-58e0b6de970c","Type":"ContainerStarted","Data":"5da49ccb06d02628714425d8c9e17c9d4918c4417ab07de44427438749705854"}
Sep 15 09:11:59.533340 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:11:59.533282    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:02.522445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:02.522409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk" event={"ID":"e3b9355e-a720-4212-881d-58e0b6de970c","Type":"ContainerStarted","Data":"011e84b91c9d053ccc03cdc35d80802ab93d764afc409a641532302b85ec9a4f"}
Sep 15 09:12:03.527918 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:03.527805    2571 generic.go:356] "Generic (PLEG): container finished" podID="e3b9355e-a720-4212-881d-58e0b6de970c" containerID="011e84b91c9d053ccc03cdc35d80802ab93d764afc409a641532302b85ec9a4f" exitCode=0
Sep 15 09:12:03.527918 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:03.527892    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk" event={"ID":"e3b9355e-a720-4212-881d-58e0b6de970c","Type":"ContainerDied","Data":"011e84b91c9d053ccc03cdc35d80802ab93d764afc409a641532302b85ec9a4f"}
Sep 15 09:12:04.534540 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:04.534456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk" event={"ID":"e3b9355e-a720-4212-881d-58e0b6de970c","Type":"ContainerStarted","Data":"5d127e6a7a5cd9182008ac6115c1518059d5871326aba6c6a4341943b3c7d8c4"}
Sep 15 09:12:04.534981 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:04.534569    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:12:04.555231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:04.555179    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk" podStartSLOduration=1.641158528 podStartE2EDuration="6.555158875s" totalImagesPullingTime="3.903576133s" totalInitContainerRuntime="1.010424214s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:11:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:12:04.553603434 +0000 UTC m=+317.012892975" watchObservedRunningTime="2026-09-15 09:12:04.555158875 +0000 UTC m=+317.014448417"
Sep 15 09:12:05.492038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:05.492003    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f667c6f8-66z7p"
Sep 15 09:12:15.541878 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:15.541843    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77dcfd7f98-6jqhk"
Sep 15 09:12:17.690956 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:17.690915    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:17.798585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:17.798539    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:18.200873 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.200818    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:18.283562 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.283524    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"]
Sep 15 09:12:18.286614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.286590    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.289615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.289598    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rcndh\""
Sep 15 09:12:18.289694 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.289678    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 09:12:18.290594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.290580    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 09:12:18.303087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.303062    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"]
Sep 15 09:12:18.377971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.377930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mktw\" (UniqueName: \"kubernetes.io/projected/68dc7d0e-069a-401c-91f1-d0fc776c4569-kube-api-access-7mktw\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.378146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.378082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68dc7d0e-069a-401c-91f1-d0fc776c4569-tls-certs\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.378146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.378115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc7d0e-069a-401c-91f1-d0fc776c4569-tmp\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.478639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.478547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68dc7d0e-069a-401c-91f1-d0fc776c4569-tls-certs\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.478639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.478585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc7d0e-069a-401c-91f1-d0fc776c4569-tmp\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.478639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.478602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mktw\" (UniqueName: \"kubernetes.io/projected/68dc7d0e-069a-401c-91f1-d0fc776c4569-kube-api-access-7mktw\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.480913 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.480880    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc7d0e-069a-401c-91f1-d0fc776c4569-tmp\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.481124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.481104    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68dc7d0e-069a-401c-91f1-d0fc776c4569-tls-certs\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.493028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.492997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mktw\" (UniqueName: \"kubernetes.io/projected/68dc7d0e-069a-401c-91f1-d0fc776c4569-kube-api-access-7mktw\") pod \"kube-auth-proxy-5d5b88c986-7p5bl\" (UID: \"68dc7d0e-069a-401c-91f1-d0fc776c4569\") " pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.502561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.502531    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:18.596138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.596108    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"
Sep 15 09:12:18.873870 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:18.873844    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl"]
Sep 15 09:12:18.876460 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:12:18.876420    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dc7d0e_069a_401c_91f1_d0fc776c4569.slice/crio-43b307e43cc58ce25d7400202475bab64ea9d47b1bd9a394c01989187bd846f9 WatchSource:0}: Error finding container 43b307e43cc58ce25d7400202475bab64ea9d47b1bd9a394c01989187bd846f9: Status 404 returned error can't find the container with id 43b307e43cc58ce25d7400202475bab64ea9d47b1bd9a394c01989187bd846f9
Sep 15 09:12:19.605124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:19.605077    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl" event={"ID":"68dc7d0e-069a-401c-91f1-d0fc776c4569","Type":"ContainerStarted","Data":"43b307e43cc58ce25d7400202475bab64ea9d47b1bd9a394c01989187bd846f9"}
Sep 15 09:12:19.635488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:19.635455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:19.732672 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:19.732622    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:20.631326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:20.631285    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:22.621918 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:22.621880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl" event={"ID":"68dc7d0e-069a-401c-91f1-d0fc776c4569","Type":"ContainerStarted","Data":"34de1077dfebd04023ca9fc57ab6e83c40c235e4eb3094820ca36614910527b3"}
Sep 15 09:12:22.639923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:22.639872    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d5b88c986-7p5bl" podStartSLOduration=1.855129818 podStartE2EDuration="4.639854473s" totalImagesPullingTime="2.784724655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:12:22.639127155 +0000 UTC m=+335.098416697" watchObservedRunningTime="2026-09-15 09:12:22.639854473 +0000 UTC m=+335.099144013"
Sep 15 09:12:24.437179 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:24.437120    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:24.535703 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:24.535671    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:25.110109 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:25.110073    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:25.219020 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:25.218965    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:28.641586 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:28.641556    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:28.742765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:28.742730    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:28.843517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:28.843463    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:29.075565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:29.075472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:29.177105 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:29.177034    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:29.258679 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:29.258646    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.142356 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.142316    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.278390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.278357    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.366736 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.366704    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.477025 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.476942    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.617573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.617539    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.722319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.722256    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:12:38.819665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:12:38.819552    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:24.921514 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:24.921478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:24.973770 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:24.973735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:26.874324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:26.874272    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:26.974980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:26.974938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:29.864977 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:29.864942    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:29.988103 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:29.988061    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:31.886037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:31.885978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:43.195846 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:43.195778    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:43.292110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:43.292071    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:43.356908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:43.356879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:45.222408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:45.222371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:45.983795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:45.983749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:49.209809 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:49.209774    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:49.440342 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:49.440284    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:49.550303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:49.550212    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:51.189894 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:51.189857    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:51.454146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:51.454058    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:51.581304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:51.581270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:52.200512 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:52.200479    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:52.469430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:52.469344    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:55.599327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:55.599290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:56.624967 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:56.624928    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:58.591036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:58.591006    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:58.691124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:58.691083    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:58.992663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:58.992584    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:13:59.072880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:13:59.072846    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:02.280493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:02.280461    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:02.565425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:02.565341    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:03.392681 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:03.392641    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:03.468994 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:03.468958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:07.989735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:07.989703    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:08.046425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:08.046388    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:08.592416 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:08.592379    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:08.692462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:08.692423    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:09.202558 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:09.202520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:09.273541 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:09.273506    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:12.204627 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.204550    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"]
Sep 15 09:14:12.207951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.207932    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.212072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.212051    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 09:14:12.212203 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.212094    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mbtvw\""
Sep 15 09:14:12.212203 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.212117    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 09:14:12.225622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.225591    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"]
Sep 15 09:14:12.362420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.362377    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdvv\" (UniqueName: \"kubernetes.io/projected/f117bb89-e03e-41bb-a476-a1abfb41e3d3-kube-api-access-xqdvv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.362597 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.362445    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f117bb89-e03e-41bb-a476-a1abfb41e3d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.463647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.463563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdvv\" (UniqueName: \"kubernetes.io/projected/f117bb89-e03e-41bb-a476-a1abfb41e3d3-kube-api-access-xqdvv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.463647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.463617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f117bb89-e03e-41bb-a476-a1abfb41e3d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.463967 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.463950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f117bb89-e03e-41bb-a476-a1abfb41e3d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.489394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.489364    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdvv\" (UniqueName: \"kubernetes.io/projected/f117bb89-e03e-41bb-a476-a1abfb41e3d3-kube-api-access-xqdvv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.518194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.518154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:12.809941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:12.809916    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"]
Sep 15 09:14:12.811155 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:14:12.811128    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf117bb89_e03e_41bb_a476_a1abfb41e3d3.slice/crio-1cebe87575646bf3d397a99c06f669942539681eb007ae7827005bd6ea299087 WatchSource:0}: Error finding container 1cebe87575646bf3d397a99c06f669942539681eb007ae7827005bd6ea299087: Status 404 returned error can't find the container with id 1cebe87575646bf3d397a99c06f669942539681eb007ae7827005bd6ea299087
Sep 15 09:14:13.107924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:13.107888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d" event={"ID":"f117bb89-e03e-41bb-a476-a1abfb41e3d3","Type":"ContainerStarted","Data":"1cebe87575646bf3d397a99c06f669942539681eb007ae7827005bd6ea299087"}
Sep 15 09:14:13.319228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:13.319190    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"]
Sep 15 09:14:13.331597 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:13.331569    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"]
Sep 15 09:14:16.263582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:16.263537    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:16.356591 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:16.356551    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:17.131856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.131767    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" containerName="manager" containerID="cri-o://bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6" gracePeriod=2
Sep 15 09:14:17.351920 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.351893    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:17.354256 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:14:17.354225    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" is forbidden: User \"system:node:ip-10-0-134-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-28.ec2.internal' and this object" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:17.410633 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.410591    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqdvv\" (UniqueName: \"kubernetes.io/projected/f117bb89-e03e-41bb-a476-a1abfb41e3d3-kube-api-access-xqdvv\") pod \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") "
Sep 15 09:14:17.410779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.410653    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f117bb89-e03e-41bb-a476-a1abfb41e3d3-extensions-socket-volume\") pod \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\" (UID: \"f117bb89-e03e-41bb-a476-a1abfb41e3d3\") "
Sep 15 09:14:17.410970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.410935    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117bb89-e03e-41bb-a476-a1abfb41e3d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f117bb89-e03e-41bb-a476-a1abfb41e3d3" (UID: "f117bb89-e03e-41bb-a476-a1abfb41e3d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:14:17.412888 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.412868    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f117bb89-e03e-41bb-a476-a1abfb41e3d3-kube-api-access-xqdvv" (OuterVolumeSpecName: "kube-api-access-xqdvv") pod "f117bb89-e03e-41bb-a476-a1abfb41e3d3" (UID: "f117bb89-e03e-41bb-a476-a1abfb41e3d3"). InnerVolumeSpecName "kube-api-access-xqdvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:14:17.511602 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.511498    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqdvv\" (UniqueName: \"kubernetes.io/projected/f117bb89-e03e-41bb-a476-a1abfb41e3d3-kube-api-access-xqdvv\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:14:17.511602 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:17.511533    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f117bb89-e03e-41bb-a476-a1abfb41e3d3-extensions-socket-volume\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:14:18.053500 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:14:18.053455    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" is forbidden: User \"system:node:ip-10-0-134-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-28.ec2.internal' and this object" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:18.054853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.054811    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" path="/var/lib/kubelet/pods/f117bb89-e03e-41bb-a476-a1abfb41e3d3/volumes"
Sep 15 09:14:18.139737 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.139701    2571 generic.go:356] "Generic (PLEG): container finished" podID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" containerID="bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6" exitCode=2
Sep 15 09:14:18.139933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.139741    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:18.139933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.139761    2571 scope.go:117] "RemoveContainer" containerID="bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6"
Sep 15 09:14:18.144201 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:14:18.144173    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-gvh9d\" is forbidden: User \"system:node:ip-10-0-134-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-28.ec2.internal' and this object" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gvh9d"
Sep 15 09:14:18.148019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.147995    2571 scope.go:117] "RemoveContainer" containerID="bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6"
Sep 15 09:14:18.148326 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:14:18.148301    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6\": container with ID starting with bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6 not found: ID does not exist" containerID="bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6"
Sep 15 09:14:18.148453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.148340    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6"} err="failed to get container status \"bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6\": rpc error: code = NotFound desc = could not find container \"bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6\": container with ID starting with bae5fb715a8e13ac36d90cf6f86fd1788f37d849a66e4ecaf494a831a159f5b6 not found: ID does not exist"
Sep 15 09:14:18.498815 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.498777    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:18.602253 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:18.602211    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:21.587641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:21.587600    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:21.647513 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:21.647478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:22.197684 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:22.197649    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:22.292889 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:22.292852    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:25.604699 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:25.604666    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:25.708414 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:25.708380    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:29.349559 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:29.349519    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:29.446667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:29.446625    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:29.547998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:29.547957    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:35.821189 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:35.821148    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:35.919689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:35.919655    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:41.875760 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.875719    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"]
Sep 15 09:14:41.876338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.876105    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" containerName="manager"
Sep 15 09:14:41.876338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.876120    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" containerName="manager"
Sep 15 09:14:41.876338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.876173    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f117bb89-e03e-41bb-a476-a1abfb41e3d3" containerName="manager"
Sep 15 09:14:41.878980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.878961    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:41.881326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.881299    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 09:14:41.881475 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.881312    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58x5m\""
Sep 15 09:14:41.889457 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.889432    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"]
Sep 15 09:14:41.900010 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:41.899980    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:42.026370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59fb6b2-5998-411a-9fc6-cd26950366a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026374    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwhf\" (UniqueName: \"kubernetes.io/projected/f59fb6b2-5998-411a-9fc6-cd26950366a0-kube-api-access-pgwhf\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026598 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026598 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026476    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026598 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026507    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026598 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026598 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.026783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.026648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127480    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59fb6b2-5998-411a-9fc6-cd26950366a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127614    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwhf\" (UniqueName: \"kubernetes.io/projected/f59fb6b2-5998-411a-9fc6-cd26950366a0-kube-api-access-pgwhf\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.127941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.128150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.127957    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.128150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.128128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.128225 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.128166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.128371 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.128340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.128504 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.128424    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f59fb6b2-5998-411a-9fc6-cd26950366a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.130088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.130069    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.130346 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.130325    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.131934 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.131912    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:42.138192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.138172    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f59fb6b2-5998-411a-9fc6-cd26950366a0-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.141045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.141027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwhf\" (UniqueName: \"kubernetes.io/projected/f59fb6b2-5998-411a-9fc6-cd26950366a0-kube-api-access-pgwhf\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cxzvg\" (UID: \"f59fb6b2-5998-411a-9fc6-cd26950366a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.189563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.189522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:42.465801 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:42.465743    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"]
Sep 15 09:14:42.468534 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:14:42.468501    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59fb6b2_5998_411a_9fc6_cd26950366a0.slice/crio-ce1f382e7736bb7bfbc2cb83ba2eebc0ed1216a3a0e0e8b3ec4f060924e67a3b WatchSource:0}: Error finding container ce1f382e7736bb7bfbc2cb83ba2eebc0ed1216a3a0e0e8b3ec4f060924e67a3b: Status 404 returned error can't find the container with id ce1f382e7736bb7bfbc2cb83ba2eebc0ed1216a3a0e0e8b3ec4f060924e67a3b
Sep 15 09:14:43.256640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:43.256599    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" event={"ID":"f59fb6b2-5998-411a-9fc6-cd26950366a0","Type":"ContainerStarted","Data":"ce1f382e7736bb7bfbc2cb83ba2eebc0ed1216a3a0e0e8b3ec4f060924e67a3b"}
Sep 15 09:14:45.067049 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:45.067012    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:14:45.067309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:45.067091    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:14:45.067309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:45.067118    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:14:45.269330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:45.269289    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" event={"ID":"f59fb6b2-5998-411a-9fc6-cd26950366a0","Type":"ContainerStarted","Data":"92c17b1e6cdc88e2ff7ea7569dc9ef7ddd572f925726428f45c4eca4ac06a932"}
Sep 15 09:14:45.291444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:45.291377    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" podStartSLOduration=1.6950710679999998 podStartE2EDuration="4.291350325s" totalImagesPullingTime="2.596279257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:45.288781146 +0000 UTC m=+477.748070688" watchObservedRunningTime="2026-09-15 09:14:45.291350325 +0000 UTC m=+477.750639870"
Sep 15 09:14:46.190093 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:46.190051    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:46.194948 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:46.194918    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:46.274980 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:46.274949    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:46.276170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:46.276148    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:14:57.402704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.402669    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:57.511151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.511113    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:57.567560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.567525    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:57.766729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.766640    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:14:57.771477 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.771450    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:57.773766 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.773740    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zpms4\""
Sep 15 09:14:57.773900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.773773    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 09:14:57.774954 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.774735    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 09:14:57.774954 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.774736    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 09:14:57.777182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.777158    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:14:57.805546 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.805506    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:14:57.874639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.874596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrl6\" (UniqueName: \"kubernetes.io/projected/fd294ba5-6ad4-49a3-8ed2-e08eea8689f3-kube-api-access-mwrl6\") pod \"limitador-limitador-84994f7ffc-fhrnp\" (UID: \"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:57.874805 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.874722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd294ba5-6ad4-49a3-8ed2-e08eea8689f3-config-file\") pod \"limitador-limitador-84994f7ffc-fhrnp\" (UID: \"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:57.975331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.975293    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd294ba5-6ad4-49a3-8ed2-e08eea8689f3-config-file\") pod \"limitador-limitador-84994f7ffc-fhrnp\" (UID: \"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:57.975493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.975340    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrl6\" (UniqueName: \"kubernetes.io/projected/fd294ba5-6ad4-49a3-8ed2-e08eea8689f3-kube-api-access-mwrl6\") pod \"limitador-limitador-84994f7ffc-fhrnp\" (UID: \"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:57.975974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.975955    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fd294ba5-6ad4-49a3-8ed2-e08eea8689f3-config-file\") pod \"limitador-limitador-84994f7ffc-fhrnp\" (UID: \"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:57.983404 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:57.983377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrl6\" (UniqueName: \"kubernetes.io/projected/fd294ba5-6ad4-49a3-8ed2-e08eea8689f3-kube-api-access-mwrl6\") pod \"limitador-limitador-84994f7ffc-fhrnp\" (UID: \"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:58.083160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.083126    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:58.233517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.233479    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:58.333586 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.333489    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:14:58.358166 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.358134    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:14:58.361995 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:14:58.361969    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd294ba5_6ad4_49a3_8ed2_e08eea8689f3.slice/crio-a973827b303868550062fc1f6c2a26308bdb1eba6463ca875b2473221847163c WatchSource:0}: Error finding container a973827b303868550062fc1f6c2a26308bdb1eba6463ca875b2473221847163c: Status 404 returned error can't find the container with id a973827b303868550062fc1f6c2a26308bdb1eba6463ca875b2473221847163c
Sep 15 09:14:58.418812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.418778    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-5m9bp"]
Sep 15 09:14:58.423317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.423297    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:14:58.425940 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.425918    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xk55v\""
Sep 15 09:14:58.429963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.429938    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5m9bp"]
Sep 15 09:14:58.580870 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.580809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3-kube-api-access-rnmh6\") pod \"authorino-59c974bd48-5m9bp\" (UID: \"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3\") " pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:14:58.682332 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.682244    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3-kube-api-access-rnmh6\") pod \"authorino-59c974bd48-5m9bp\" (UID: \"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3\") " pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:14:58.691101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.691077    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3-kube-api-access-rnmh6\") pod \"authorino-59c974bd48-5m9bp\" (UID: \"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3\") " pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:14:58.732958 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:58.732926    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:14:59.002443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:59.002361    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5m9bp"]
Sep 15 09:14:59.005903 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:14:59.005872    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e7b399c_f97a_4c5a_8dc4_fe6292acf2a3.slice/crio-8a11fb34e083642e3f1f7830f319b72fdb0e85b96fdfb7ef4a2295e6681b02bd WatchSource:0}: Error finding container 8a11fb34e083642e3f1f7830f319b72fdb0e85b96fdfb7ef4a2295e6681b02bd: Status 404 returned error can't find the container with id 8a11fb34e083642e3f1f7830f319b72fdb0e85b96fdfb7ef4a2295e6681b02bd
Sep 15 09:14:59.339898 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:59.339856    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5m9bp" event={"ID":"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3","Type":"ContainerStarted","Data":"8a11fb34e083642e3f1f7830f319b72fdb0e85b96fdfb7ef4a2295e6681b02bd"}
Sep 15 09:14:59.341359 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:59.341331    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp" event={"ID":"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3","Type":"ContainerStarted","Data":"954be5536281e7bb567b8b344b8a074bea9f23e25f5f4f951353f4c3df39a106"}
Sep 15 09:14:59.341487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:59.341386    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp" event={"ID":"fd294ba5-6ad4-49a3-8ed2-e08eea8689f3","Type":"ContainerStarted","Data":"a973827b303868550062fc1f6c2a26308bdb1eba6463ca875b2473221847163c"}
Sep 15 09:14:59.341487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:59.341415    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:14:59.358752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:14:59.358695    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp" podStartSLOduration=1.4659953909999999 podStartE2EDuration="2.358680026s" totalImagesPullingTime="892.684635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:14:59.357642196 +0000 UTC m=+491.816931751" watchObservedRunningTime="2026-09-15 09:14:59.358680026 +0000 UTC m=+491.817969567"
Sep 15 09:15:01.353327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:01.353287    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5m9bp" event={"ID":"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3","Type":"ContainerStarted","Data":"79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0"}
Sep 15 09:15:01.368304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:01.368246    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-5m9bp" podStartSLOduration=1.954451058 podStartE2EDuration="3.368229419s" totalImagesPullingTime="1.413778361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:01.367266841 +0000 UTC m=+493.826556385" watchObservedRunningTime="2026-09-15 09:15:01.368229419 +0000 UTC m=+493.827518961"
Sep 15 09:15:10.348625 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:10.348585    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"
Sep 15 09:15:12.717915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:12.717869    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:12.817593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:12.817559    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:13.241134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:13.241095    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:23.012484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.012434    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:23.225842 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.225790    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4hdkk"]
Sep 15 09:15:23.230690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.230667    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:23.236449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.236410    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4hdkk"]
Sep 15 09:15:23.290904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.290776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8q4g\" (UniqueName: \"kubernetes.io/projected/525fa77d-dc00-4f5e-9c76-e497212e443d-kube-api-access-d8q4g\") pod \"authorino-557f876f69-4hdkk\" (UID: \"525fa77d-dc00-4f5e-9c76-e497212e443d\") " pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:23.391666 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.391625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8q4g\" (UniqueName: \"kubernetes.io/projected/525fa77d-dc00-4f5e-9c76-e497212e443d-kube-api-access-d8q4g\") pod \"authorino-557f876f69-4hdkk\" (UID: \"525fa77d-dc00-4f5e-9c76-e497212e443d\") " pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:23.399289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.399260    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8q4g\" (UniqueName: \"kubernetes.io/projected/525fa77d-dc00-4f5e-9c76-e497212e443d-kube-api-access-d8q4g\") pod \"authorino-557f876f69-4hdkk\" (UID: \"525fa77d-dc00-4f5e-9c76-e497212e443d\") " pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:23.454738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.454701    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:23.466892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.466861    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4hdkk"]
Sep 15 09:15:23.467137 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.467125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:23.494231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.494194    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-747d7b64b7-7d8cb"]
Sep 15 09:15:23.497188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.497160    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.499339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.499310    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 09:15:23.504887 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.504861    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747d7b64b7-7d8cb"]
Sep 15 09:15:23.593554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.593516    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdjp\" (UniqueName: \"kubernetes.io/projected/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-kube-api-access-4vdjp\") pod \"authorino-747d7b64b7-7d8cb\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") " pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.593742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.593644    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-tls-cert\") pod \"authorino-747d7b64b7-7d8cb\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") " pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.694164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.694121    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdjp\" (UniqueName: \"kubernetes.io/projected/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-kube-api-access-4vdjp\") pod \"authorino-747d7b64b7-7d8cb\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") " pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.694347 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.694200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-tls-cert\") pod \"authorino-747d7b64b7-7d8cb\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") " pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.696610 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.696584    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-tls-cert\") pod \"authorino-747d7b64b7-7d8cb\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") " pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.706208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.706180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdjp\" (UniqueName: \"kubernetes.io/projected/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-kube-api-access-4vdjp\") pod \"authorino-747d7b64b7-7d8cb\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") " pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:23.739760 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.739729    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4hdkk"]
Sep 15 09:15:23.742381 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:15:23.742345    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod525fa77d_dc00_4f5e_9c76_e497212e443d.slice/crio-8ab81e3ca5a279708e3f77a72a0c1206c7bd097f88328b0b15095dc199d03b03 WatchSource:0}: Error finding container 8ab81e3ca5a279708e3f77a72a0c1206c7bd097f88328b0b15095dc199d03b03: Status 404 returned error can't find the container with id 8ab81e3ca5a279708e3f77a72a0c1206c7bd097f88328b0b15095dc199d03b03
Sep 15 09:15:23.808629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:23.808588    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:15:24.077635 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.077527    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747d7b64b7-7d8cb"]
Sep 15 09:15:24.080352 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:15:24.080327    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc959cdd_11f1_4c7a_a370_14a8e8110dc9.slice/crio-478e7486b0f490c3f9be3b734561641a7bb554f6bebf033141d9a3c162f9971b WatchSource:0}: Error finding container 478e7486b0f490c3f9be3b734561641a7bb554f6bebf033141d9a3c162f9971b: Status 404 returned error can't find the container with id 478e7486b0f490c3f9be3b734561641a7bb554f6bebf033141d9a3c162f9971b
Sep 15 09:15:24.457171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.457136    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d7b64b7-7d8cb" event={"ID":"dc959cdd-11f1-4c7a-a370-14a8e8110dc9","Type":"ContainerStarted","Data":"478e7486b0f490c3f9be3b734561641a7bb554f6bebf033141d9a3c162f9971b"}
Sep 15 09:15:24.459188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.459156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-4hdkk" event={"ID":"525fa77d-dc00-4f5e-9c76-e497212e443d","Type":"ContainerStarted","Data":"d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43"}
Sep 15 09:15:24.459298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.459204    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-4hdkk" event={"ID":"525fa77d-dc00-4f5e-9c76-e497212e443d","Type":"ContainerStarted","Data":"8ab81e3ca5a279708e3f77a72a0c1206c7bd097f88328b0b15095dc199d03b03"}
Sep 15 09:15:24.459298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.459243    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-4hdkk" podUID="525fa77d-dc00-4f5e-9c76-e497212e443d" containerName="authorino" containerID="cri-o://d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43" gracePeriod=30
Sep 15 09:15:24.473099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.473057    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-4hdkk" podStartSLOduration=0.841335011 podStartE2EDuration="1.473040153s" totalImagesPullingTime="631.705142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:24.472190769 +0000 UTC m=+516.931480310" watchObservedRunningTime="2026-09-15 09:15:24.473040153 +0000 UTC m=+516.932329694"
Sep 15 09:15:24.708449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.708415    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:24.805561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.805521    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8q4g\" (UniqueName: \"kubernetes.io/projected/525fa77d-dc00-4f5e-9c76-e497212e443d-kube-api-access-d8q4g\") pod \"525fa77d-dc00-4f5e-9c76-e497212e443d\" (UID: \"525fa77d-dc00-4f5e-9c76-e497212e443d\") "
Sep 15 09:15:24.807628 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.807600    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525fa77d-dc00-4f5e-9c76-e497212e443d-kube-api-access-d8q4g" (OuterVolumeSpecName: "kube-api-access-d8q4g") pod "525fa77d-dc00-4f5e-9c76-e497212e443d" (UID: "525fa77d-dc00-4f5e-9c76-e497212e443d"). InnerVolumeSpecName "kube-api-access-d8q4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:15:24.906941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:24.906852    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8q4g\" (UniqueName: \"kubernetes.io/projected/525fa77d-dc00-4f5e-9c76-e497212e443d-kube-api-access-d8q4g\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:15:25.465898 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.465863    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d7b64b7-7d8cb" event={"ID":"dc959cdd-11f1-4c7a-a370-14a8e8110dc9","Type":"ContainerStarted","Data":"69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442"}
Sep 15 09:15:25.467111 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.467087    2571 generic.go:356] "Generic (PLEG): container finished" podID="525fa77d-dc00-4f5e-9c76-e497212e443d" containerID="d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43" exitCode=2
Sep 15 09:15:25.467199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.467112    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4hdkk"
Sep 15 09:15:25.467199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.467142    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-4hdkk" event={"ID":"525fa77d-dc00-4f5e-9c76-e497212e443d","Type":"ContainerDied","Data":"d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43"}
Sep 15 09:15:25.467199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.467169    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-4hdkk" event={"ID":"525fa77d-dc00-4f5e-9c76-e497212e443d","Type":"ContainerDied","Data":"8ab81e3ca5a279708e3f77a72a0c1206c7bd097f88328b0b15095dc199d03b03"}
Sep 15 09:15:25.467199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.467185    2571 scope.go:117] "RemoveContainer" containerID="d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43"
Sep 15 09:15:25.476002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.475983    2571 scope.go:117] "RemoveContainer" containerID="d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43"
Sep 15 09:15:25.476295 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:15:25.476276    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43\": container with ID starting with d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43 not found: ID does not exist" containerID="d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43"
Sep 15 09:15:25.476342 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.476304    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43"} err="failed to get container status \"d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43\": rpc error: code = NotFound desc = could not find container \"d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43\": container with ID starting with d0330486cb7dbaf4c17c74393f129cfe44d3187e5582bca0701eb53dcb1ebe43 not found: ID does not exist"
Sep 15 09:15:25.482584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.482539    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-747d7b64b7-7d8cb" podStartSLOduration=2.143241789 podStartE2EDuration="2.482524553s" totalImagesPullingTime="339.282764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:15:25.480967121 +0000 UTC m=+517.940256662" watchObservedRunningTime="2026-09-15 09:15:25.482524553 +0000 UTC m=+517.941814094"
Sep 15 09:15:25.495910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.495882    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4hdkk"]
Sep 15 09:15:25.499689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.499661    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4hdkk"]
Sep 15 09:15:25.509205 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.509179    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5m9bp"]
Sep 15 09:15:25.509376 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.509355    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-5m9bp" podUID="1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" containerName="authorino" containerID="cri-o://79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0" gracePeriod=30
Sep 15 09:15:25.748601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.748574    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:15:25.813548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.813512    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3-kube-api-access-rnmh6\") pod \"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3\" (UID: \"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3\") "
Sep 15 09:15:25.815623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.815591    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3-kube-api-access-rnmh6" (OuterVolumeSpecName: "kube-api-access-rnmh6") pod "1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" (UID: "1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3"). InnerVolumeSpecName "kube-api-access-rnmh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:15:25.915122 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:25.915082    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmh6\" (UniqueName: \"kubernetes.io/projected/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3-kube-api-access-rnmh6\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:15:26.054640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.054558    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525fa77d-dc00-4f5e-9c76-e497212e443d" path="/var/lib/kubelet/pods/525fa77d-dc00-4f5e-9c76-e497212e443d/volumes"
Sep 15 09:15:26.473415 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.473377    2571 generic.go:356] "Generic (PLEG): container finished" podID="1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" containerID="79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0" exitCode=0
Sep 15 09:15:26.473867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.473419    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5m9bp"
Sep 15 09:15:26.473867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.473461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5m9bp" event={"ID":"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3","Type":"ContainerDied","Data":"79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0"}
Sep 15 09:15:26.473867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.473499    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5m9bp" event={"ID":"1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3","Type":"ContainerDied","Data":"8a11fb34e083642e3f1f7830f319b72fdb0e85b96fdfb7ef4a2295e6681b02bd"}
Sep 15 09:15:26.473867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.473520    2571 scope.go:117] "RemoveContainer" containerID="79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0"
Sep 15 09:15:26.481707 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.481686    2571 scope.go:117] "RemoveContainer" containerID="79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0"
Sep 15 09:15:26.482002 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:15:26.481985    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0\": container with ID starting with 79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0 not found: ID does not exist" containerID="79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0"
Sep 15 09:15:26.482060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.482009    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0"} err="failed to get container status \"79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0\": rpc error: code = NotFound desc = could not find container \"79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0\": container with ID starting with 79c9d1d6385b11ca93b5700d76f3a34d2badd0dc063f5be79a517a26ac01dcb0 not found: ID does not exist"
Sep 15 09:15:26.488928 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.488891    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5m9bp"]
Sep 15 09:15:26.490432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:26.490411    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5m9bp"]
Sep 15 09:15:27.308270 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:27.308237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:28.055017 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:28.054981    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" path="/var/lib/kubelet/pods/1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3/volumes"
Sep 15 09:15:30.222467 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:30.222414    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:30.321291 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:30.321254    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:30.378041 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:30.378004    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:30.511559 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:30.511474    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:33.543265 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:33.543235    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:33.664875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:33.664811    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:15:55.662284 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:15:55.662248    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:03.292482 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:03.292438    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:03.392914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:03.392875    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:04.885363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:04.885329    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:04.993326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:04.993290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:05.060100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:05.060040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:05.335171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:05.335133    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:05.431794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:05.431758    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:05.502418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:05.502386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:05.651237 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:05.651138    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:16:14.696241 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:14.696208    2571 generic.go:356] "Generic (PLEG): container finished" podID="f59fb6b2-5998-411a-9fc6-cd26950366a0" containerID="92c17b1e6cdc88e2ff7ea7569dc9ef7ddd572f925726428f45c4eca4ac06a932" exitCode=0
Sep 15 09:16:14.696705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:14.696278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" event={"ID":"f59fb6b2-5998-411a-9fc6-cd26950366a0","Type":"ContainerDied","Data":"92c17b1e6cdc88e2ff7ea7569dc9ef7ddd572f925726428f45c4eca4ac06a932"}
Sep 15 09:16:14.696705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:14.696676    2571 scope.go:117] "RemoveContainer" containerID="92c17b1e6cdc88e2ff7ea7569dc9ef7ddd572f925726428f45c4eca4ac06a932"
Sep 15 09:16:14.697669 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:14.697636    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:16:14.697753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:14.697706    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:16:14.697753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:14.697737    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:16:15.703104 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:15.703072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" event={"ID":"f59fb6b2-5998-411a-9fc6-cd26950366a0","Type":"ContainerStarted","Data":"370d8f635a5f8e44257866b178ef9181879266674b07580ef30212877191a9ab"}
Sep 15 09:16:16.005535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:16.005443    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:16.101453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:16.101419    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:16.190387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:16.190352    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:16:16.191528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:16.191508    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cxzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 15 09:16:16.191609 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:16.191569    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" podUID="f59fb6b2-5998-411a-9fc6-cd26950366a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 15 09:16:16.275059 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:16.274961    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:16:17.189910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:17.189869    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cxzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 15 09:16:17.190434 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:17.189933    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" podUID="f59fb6b2-5998-411a-9fc6-cd26950366a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 15 09:16:18.190265 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:18.190229    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cxzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 15 09:16:18.190643 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:18.190284    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" podUID="f59fb6b2-5998-411a-9fc6-cd26950366a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 15 09:16:19.190627 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:19.190586    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cxzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 15 09:16:19.191060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:19.190657    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" podUID="f59fb6b2-5998-411a-9fc6-cd26950366a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 15 09:16:20.190205 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:20.190165    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cxzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 15 09:16:20.190411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:20.190225    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg" podUID="f59fb6b2-5998-411a-9fc6-cd26950366a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 15 09:16:21.194463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:21.194429    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:16:21.195421 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:21.195398    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cxzvg"
Sep 15 09:16:34.512735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:34.512694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:35.016708 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:35.016671    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:16:47.975188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:47.975161    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:16:47.975670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:47.975448    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:16:50.835732 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:50.835694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:51.602871 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:51.602808    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:16:58.242498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:58.242459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:16:59.311191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:16:59.311154    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:17:19.447246 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:19.447207    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:17:20.600958 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:20.600923    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:17:26.938645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:26.938611    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:17:27.903612 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:27.903575    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:17:41.210844 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.210799    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"]
Sep 15 09:17:41.211274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.211150    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" containerName="authorino"
Sep 15 09:17:41.211274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.211162    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" containerName="authorino"
Sep 15 09:17:41.211274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.211179    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525fa77d-dc00-4f5e-9c76-e497212e443d" containerName="authorino"
Sep 15 09:17:41.211274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.211184    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="525fa77d-dc00-4f5e-9c76-e497212e443d" containerName="authorino"
Sep 15 09:17:41.211274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.211259    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="525fa77d-dc00-4f5e-9c76-e497212e443d" containerName="authorino"
Sep 15 09:17:41.211274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.211268    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7b399c-f97a-4c5a-8dc4-fe6292acf2a3" containerName="authorino"
Sep 15 09:17:41.214361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.214344    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.217033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.217008    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 15 09:17:41.217264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.217243    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c6slb\""
Sep 15 09:17:41.217357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.217053    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 09:17:41.217420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.217056    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 09:17:41.223455 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.223431    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"]
Sep 15 09:17:41.249165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.249128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:17:41.388544 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.388502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.388747 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.388575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f4329-cd5b-453f-83b1-89b180df9c0d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.388747 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.388600    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.388747 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.388656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lntg7\" (UniqueName: \"kubernetes.io/projected/0b0f4329-cd5b-453f-83b1-89b180df9c0d-kube-api-access-lntg7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.388747 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.388694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.388960 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.388775    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.489756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.489654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.489756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.489721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f4329-cd5b-453f-83b1-89b180df9c0d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.489756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.489740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.490092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.489771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lntg7\" (UniqueName: \"kubernetes.io/projected/0b0f4329-cd5b-453f-83b1-89b180df9c0d-kube-api-access-lntg7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.490092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.489790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.490092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.489869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.490253 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.490193    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.490304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.490250    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.490304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.490266    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.492131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.492112    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0f4329-cd5b-453f-83b1-89b180df9c0d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.492307 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.492287    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0f4329-cd5b-453f-83b1-89b180df9c0d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.497933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.497908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lntg7\" (UniqueName: \"kubernetes.io/projected/0b0f4329-cd5b-453f-83b1-89b180df9c0d-kube-api-access-lntg7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdl7t\" (UID: \"0b0f4329-cd5b-453f-83b1-89b180df9c0d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.527034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.526990    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:41.799361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.799237    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"]
Sep 15 09:17:41.801666 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:17:41.801635    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0f4329_cd5b_453f_83b1_89b180df9c0d.slice/crio-dcb3979e472fc924e0452c8f26d0b9e002b5d5b76fc4726bf3e79e3e75a14692 WatchSource:0}: Error finding container dcb3979e472fc924e0452c8f26d0b9e002b5d5b76fc4726bf3e79e3e75a14692: Status 404 returned error can't find the container with id dcb3979e472fc924e0452c8f26d0b9e002b5d5b76fc4726bf3e79e3e75a14692
Sep 15 09:17:41.803022 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:41.803004    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:17:42.099792 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:42.099751    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t" event={"ID":"0b0f4329-cd5b-453f-83b1-89b180df9c0d","Type":"ContainerStarted","Data":"dcb3979e472fc924e0452c8f26d0b9e002b5d5b76fc4726bf3e79e3e75a14692"}
Sep 15 09:17:42.213320 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:42.213280    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:17:43.108266 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:43.108215    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t" event={"ID":"0b0f4329-cd5b-453f-83b1-89b180df9c0d","Type":"ContainerStarted","Data":"f2443603e5fb4f18f687c451d2b5f5697d6165f74e67872df9f2c1fc601adaa6"}
Sep 15 09:17:43.126166 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:43.126109    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t" podStartSLOduration=1.025469062 podStartE2EDuration="2.126094235s" totalImagesPullingTime="1.100625173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:17:43.12475982 +0000 UTC m=+655.584049372" watchObservedRunningTime="2026-09-15 09:17:43.126094235 +0000 UTC m=+655.585383775"
Sep 15 09:17:51.527532 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:51.527491    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:51.528072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:51.527588    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:51.540010 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:51.539981    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:52.161151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:52.161119    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdl7t"
Sep 15 09:17:53.205811 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:17:53.205779    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:18:07.109103 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.109063    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"]
Sep 15 09:18:07.112756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.112734    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.115207 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.115183    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 09:18:07.121709 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.121682    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"]
Sep 15 09:18:07.138183 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.138155    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:18:07.214793 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.214760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.214970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.214801    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f400c0e-b16e-4698-bb51-e0e08c821ebe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.214970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.214846    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdvv\" (UniqueName: \"kubernetes.io/projected/7f400c0e-b16e-4698-bb51-e0e08c821ebe-kube-api-access-4qdvv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.214970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.214934    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.215076 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.214999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.215076 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.215047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f400c0e-b16e-4698-bb51-e0e08c821ebe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316160    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdvv\" (UniqueName: \"kubernetes.io/projected/7f400c0e-b16e-4698-bb51-e0e08c821ebe-kube-api-access-4qdvv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316254    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316606 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316569    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316606 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316581    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.316718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.316638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.318514 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.318489    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f400c0e-b16e-4698-bb51-e0e08c821ebe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.318817 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.318794    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f400c0e-b16e-4698-bb51-e0e08c821ebe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.324069 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.324043    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdvv\" (UniqueName: \"kubernetes.io/projected/7f400c0e-b16e-4698-bb51-e0e08c821ebe-kube-api-access-4qdvv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q25hl\" (UID: \"7f400c0e-b16e-4698-bb51-e0e08c821ebe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.423035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.422938    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:07.696085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:07.696056    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"]
Sep 15 09:18:07.698695 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:18:07.698667    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f400c0e_b16e_4698_bb51_e0e08c821ebe.slice/crio-c44a1db2893f33930aa62b18e7d772e663e3c064607966bb3b2a312226fdb716 WatchSource:0}: Error finding container c44a1db2893f33930aa62b18e7d772e663e3c064607966bb3b2a312226fdb716: Status 404 returned error can't find the container with id c44a1db2893f33930aa62b18e7d772e663e3c064607966bb3b2a312226fdb716
Sep 15 09:18:08.222975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:08.222939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl" event={"ID":"7f400c0e-b16e-4698-bb51-e0e08c821ebe","Type":"ContainerStarted","Data":"f2c2bc87ee79111b5735fdb91025078690e57f6800212e2a0e78f2fb34d56abe"}
Sep 15 09:18:08.222975 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:08.222982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl" event={"ID":"7f400c0e-b16e-4698-bb51-e0e08c821ebe","Type":"ContainerStarted","Data":"c44a1db2893f33930aa62b18e7d772e663e3c064607966bb3b2a312226fdb716"}
Sep 15 09:18:08.240041 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:08.239977    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl" podStartSLOduration=1.058266629 podStartE2EDuration="1.23996145s" totalImagesPullingTime="181.694821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:18:08.238554982 +0000 UTC m=+680.697844523" watchObservedRunningTime="2026-09-15 09:18:08.23996145 +0000 UTC m=+680.699250991"
Sep 15 09:18:09.215583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:09.215545    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:18:17.423054 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:17.423002    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:17.423449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:17.423078    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:17.436462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:17.436425    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:18:18.285306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:18:18.285268    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q25hl"
Sep 15 09:19:07.521224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:07.521102    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:17.397505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.397468    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-748b84f758-jw6tb"]
Sep 15 09:19:17.400786 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.400763    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.409081 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.409045    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-748b84f758-jw6tb"]
Sep 15 09:19:17.569727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.569683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c6ab9-2d03-45e1-83e2-29f9e42374e2-tls-cert\") pod \"authorino-748b84f758-jw6tb\" (UID: \"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2\") " pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.569919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.569791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2fh2\" (UniqueName: \"kubernetes.io/projected/cc6c6ab9-2d03-45e1-83e2-29f9e42374e2-kube-api-access-c2fh2\") pod \"authorino-748b84f758-jw6tb\" (UID: \"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2\") " pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.671309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.671233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2fh2\" (UniqueName: \"kubernetes.io/projected/cc6c6ab9-2d03-45e1-83e2-29f9e42374e2-kube-api-access-c2fh2\") pod \"authorino-748b84f758-jw6tb\" (UID: \"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2\") " pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.671309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.671306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c6ab9-2d03-45e1-83e2-29f9e42374e2-tls-cert\") pod \"authorino-748b84f758-jw6tb\" (UID: \"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2\") " pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.673868 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.673821    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc6c6ab9-2d03-45e1-83e2-29f9e42374e2-tls-cert\") pod \"authorino-748b84f758-jw6tb\" (UID: \"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2\") " pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.679490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.679464    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2fh2\" (UniqueName: \"kubernetes.io/projected/cc6c6ab9-2d03-45e1-83e2-29f9e42374e2-kube-api-access-c2fh2\") pod \"authorino-748b84f758-jw6tb\" (UID: \"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2\") " pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.710337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.710293    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-748b84f758-jw6tb"
Sep 15 09:19:17.980347 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:17.980318    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-748b84f758-jw6tb"]
Sep 15 09:19:17.982851 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:17.982804    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc6c6ab9_2d03_45e1_83e2_29f9e42374e2.slice/crio-2fd20a743b677ac79fd76740171becfa73d74ca4db00767d46b1f1b3355a5818 WatchSource:0}: Error finding container 2fd20a743b677ac79fd76740171becfa73d74ca4db00767d46b1f1b3355a5818: Status 404 returned error can't find the container with id 2fd20a743b677ac79fd76740171becfa73d74ca4db00767d46b1f1b3355a5818
Sep 15 09:19:18.558721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:18.558684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-748b84f758-jw6tb" event={"ID":"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2","Type":"ContainerStarted","Data":"2fd20a743b677ac79fd76740171becfa73d74ca4db00767d46b1f1b3355a5818"}
Sep 15 09:19:19.567148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.567083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-748b84f758-jw6tb" event={"ID":"cc6c6ab9-2d03-45e1-83e2-29f9e42374e2","Type":"ContainerStarted","Data":"23ca83d1c0720955e90831021e3d6ccbe659c7809e2be48063181199232653a6"}
Sep 15 09:19:19.585904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.585819    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-748b84f758-jw6tb" podStartSLOduration=1.9335407770000002 podStartE2EDuration="2.585796107s" totalImagesPullingTime="652.25533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:19.583386995 +0000 UTC m=+752.042676549" watchObservedRunningTime="2026-09-15 09:19:19.585796107 +0000 UTC m=+752.045085648"
Sep 15 09:19:19.608796 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.608742    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747d7b64b7-7d8cb"]
Sep 15 09:19:19.609012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.608987    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-747d7b64b7-7d8cb" podUID="dc959cdd-11f1-4c7a-a370-14a8e8110dc9" containerName="authorino" containerID="cri-o://69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442" gracePeriod=30
Sep 15 09:19:19.863906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.863869    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:19:19.993663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.993629    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vdjp\" (UniqueName: \"kubernetes.io/projected/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-kube-api-access-4vdjp\") pod \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") "
Sep 15 09:19:19.993663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.993665    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-tls-cert\") pod \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\" (UID: \"dc959cdd-11f1-4c7a-a370-14a8e8110dc9\") "
Sep 15 09:19:19.995803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:19.995771    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-kube-api-access-4vdjp" (OuterVolumeSpecName: "kube-api-access-4vdjp") pod "dc959cdd-11f1-4c7a-a370-14a8e8110dc9" (UID: "dc959cdd-11f1-4c7a-a370-14a8e8110dc9"). InnerVolumeSpecName "kube-api-access-4vdjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:19:20.006581 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.006544    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dc959cdd-11f1-4c7a-a370-14a8e8110dc9" (UID: "dc959cdd-11f1-4c7a-a370-14a8e8110dc9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:19:20.094472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.094372    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vdjp\" (UniqueName: \"kubernetes.io/projected/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-kube-api-access-4vdjp\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:19:20.094472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.094408    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc959cdd-11f1-4c7a-a370-14a8e8110dc9-tls-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:19:20.112956 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.112913    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:20.209820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.209786    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:20.573874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.573752    2571 generic.go:356] "Generic (PLEG): container finished" podID="dc959cdd-11f1-4c7a-a370-14a8e8110dc9" containerID="69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442" exitCode=0
Sep 15 09:19:20.573874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.573820    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747d7b64b7-7d8cb"
Sep 15 09:19:20.573874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.573852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d7b64b7-7d8cb" event={"ID":"dc959cdd-11f1-4c7a-a370-14a8e8110dc9","Type":"ContainerDied","Data":"69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442"}
Sep 15 09:19:20.574423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.573906    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747d7b64b7-7d8cb" event={"ID":"dc959cdd-11f1-4c7a-a370-14a8e8110dc9","Type":"ContainerDied","Data":"478e7486b0f490c3f9be3b734561641a7bb554f6bebf033141d9a3c162f9971b"}
Sep 15 09:19:20.574423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.573929    2571 scope.go:117] "RemoveContainer" containerID="69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442"
Sep 15 09:19:20.582902 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.582882    2571 scope.go:117] "RemoveContainer" containerID="69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442"
Sep 15 09:19:20.583279 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:20.583257    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442\": container with ID starting with 69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442 not found: ID does not exist" containerID="69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442"
Sep 15 09:19:20.583340 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.583287    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442"} err="failed to get container status \"69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442\": rpc error: code = NotFound desc = could not find container \"69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442\": container with ID starting with 69e556ee325fb4695c49353f91ff34ecec268abdf249535c32bd18f80a36b442 not found: ID does not exist"
Sep 15 09:19:20.590942 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.590907    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747d7b64b7-7d8cb"]
Sep 15 09:19:20.593676 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:20.593650    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-747d7b64b7-7d8cb"]
Sep 15 09:19:21.252881 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:21.252814    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:21.352534 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:21.352489    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:22.054571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:22.054533    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc959cdd-11f1-4c7a-a370-14a8e8110dc9" path="/var/lib/kubelet/pods/dc959cdd-11f1-4c7a-a370-14a8e8110dc9/volumes"
Sep 15 09:19:29.416711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:29.416666    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:29.472382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:29.472333    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.145605 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.145572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.246067 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.246038    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.349734 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.349695    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.509039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.508962    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.686436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.686407    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.735863 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.735813    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"]
Sep 15 09:19:42.736249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.736232    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc959cdd-11f1-4c7a-a370-14a8e8110dc9" containerName="authorino"
Sep 15 09:19:42.736249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.736249    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc959cdd-11f1-4c7a-a370-14a8e8110dc9" containerName="authorino"
Sep 15 09:19:42.736398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.736312    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc959cdd-11f1-4c7a-a370-14a8e8110dc9" containerName="authorino"
Sep 15 09:19:42.740696 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.740646    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.744705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.744686    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-890f3e-openshift-default-dockercfg-54d4j\""
Sep 15 09:19:42.752738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.752714    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"]
Sep 15 09:19:42.789125 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.789049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.795238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25538817-dc63-42ee-a527-35d8b90c0207-istiod-ca-cert\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtvc\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-kube-api-access-6rtvc\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-data\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795466 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-certs\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795466 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795398    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-envoy\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795466 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-istio-token\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-credential-socket\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795536    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-socket\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.795592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.795554    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25538817-dc63-42ee-a527-35d8b90c0207-istio-podinfo\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.848950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.848916    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:42.896193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-certs\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-envoy\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-istio-token\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-credential-socket\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896272    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-socket\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25538817-dc63-42ee-a527-35d8b90c0207-istio-podinfo\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25538817-dc63-42ee-a527-35d8b90c0207-istiod-ca-cert\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtvc\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-kube-api-access-6rtvc\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-data\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-certs\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896757    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-data\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.896945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896856    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-socket\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.897004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.896969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-credential-socket\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.897112 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.897090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25538817-dc63-42ee-a527-35d8b90c0207-istiod-ca-cert\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.898788 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.898766    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25538817-dc63-42ee-a527-35d8b90c0207-istio-podinfo\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.898925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.898909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-envoy\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.903956 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.903934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-istio-token\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.904094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.904076    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtvc\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-kube-api-access-6rtvc\") pod \"e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") " pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:42.967190 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:42.967134    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:43.052802 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.052476    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:43.088025 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.087994    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:43.327168 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.327075    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"]
Sep 15 09:19:43.329936 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:43.329906    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25538817_dc63_42ee_a527_35d8b90c0207.slice/crio-6992d5b9fe3087065b48759055a58bad7c51c20e1e9d6b361c0244020cf957f1 WatchSource:0}: Error finding container 6992d5b9fe3087065b48759055a58bad7c51c20e1e9d6b361c0244020cf957f1: Status 404 returned error can't find the container with id 6992d5b9fe3087065b48759055a58bad7c51c20e1e9d6b361c0244020cf957f1
Sep 15 09:19:43.332268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.332229    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:43.332357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.332320    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:43.332396 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.332364    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:43.680893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.680857    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" event={"ID":"25538817-dc63-42ee-a527-35d8b90c0207","Type":"ContainerStarted","Data":"ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"}
Sep 15 09:19:43.681070 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.680918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" event={"ID":"25538817-dc63-42ee-a527-35d8b90c0207","Type":"ContainerStarted","Data":"6992d5b9fe3087065b48759055a58bad7c51c20e1e9d6b361c0244020cf957f1"}
Sep 15 09:19:43.702100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.702046    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podStartSLOduration=1.702027607 podStartE2EDuration="1.702027607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:43.700718979 +0000 UTC m=+776.160008544" watchObservedRunningTime="2026-09-15 09:19:43.702027607 +0000 UTC m=+776.161317149"
Sep 15 09:19:43.819044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.819010    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:43.918445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:43.918405    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:44.060751 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.060670    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:44.060914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.060756    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:44.338730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.338704    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:44.513948 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.513911    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7750-14f65c"]
Sep 15 09:19:44.517380 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.517356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:44.519879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.519860    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7g2n6\""
Sep 15 09:19:44.525775 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.525549    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7750-14f65c"]
Sep 15 09:19:44.613535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.613449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgdj4\" (UniqueName: \"kubernetes.io/projected/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1-kube-api-access-xgdj4\") pod \"test-curl-7750-14f65c\" (UID: \"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1\") " pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:44.686749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.686714    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:44.688011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.687983    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:19:44.714247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.714213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgdj4\" (UniqueName: \"kubernetes.io/projected/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1-kube-api-access-xgdj4\") pod \"test-curl-7750-14f65c\" (UID: \"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1\") " pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:44.722275 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.722243    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgdj4\" (UniqueName: \"kubernetes.io/projected/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1-kube-api-access-xgdj4\") pod \"test-curl-7750-14f65c\" (UID: \"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1\") " pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:44.829235 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:44.829200    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:45.098332 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:45.098302    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7750-14f65c"]
Sep 15 09:19:45.101084 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:45.101054    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ceed7d4_7e7e_452d_8f6b_c95c1c2277e1.slice/crio-3d5dce448d1688f08dd3a886cbcf35172162931bd0c45fd1f6dee7f974c5262f WatchSource:0}: Error finding container 3d5dce448d1688f08dd3a886cbcf35172162931bd0c45fd1f6dee7f974c5262f: Status 404 returned error can't find the container with id 3d5dce448d1688f08dd3a886cbcf35172162931bd0c45fd1f6dee7f974c5262f
Sep 15 09:19:45.692808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:45.692770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7750-14f65c" event={"ID":"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1","Type":"ContainerStarted","Data":"3d5dce448d1688f08dd3a886cbcf35172162931bd0c45fd1f6dee7f974c5262f"}
Sep 15 09:19:45.870924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:45.870888    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:46.026186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:46.026093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:46.090040 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:46.090000    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:19:46.149963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:46.149930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:47.411039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:47.410951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:47.574353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:47.574258    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:47.887325 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:47.887297    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:48.074075 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.073993    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:48.253113 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.253075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:48.392789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.392760    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:48.465471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.465434    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:48.710659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.710569    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7750-14f65c" event={"ID":"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1","Type":"ContainerStarted","Data":"852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a"}
Sep 15 09:19:48.724680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.724629    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7750-14f65c" podStartSLOduration=2.031321902 podStartE2EDuration="4.724615217s" totalImagesPullingTime="2.693293315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:48.724152853 +0000 UTC m=+781.183442394" watchObservedRunningTime="2026-09-15 09:19:48.724615217 +0000 UTC m=+781.183904748"
Sep 15 09:19:48.803066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.803026    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:48.998623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:48.998543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:49.095882 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.095848    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:49.371619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.371587    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7750-14f65c"]
Sep 15 09:19:49.376304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.376276    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7750-14f65c"]
Sep 15 09:19:49.409647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.409615    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:49.516920 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.516881    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7750-b1197e"]
Sep 15 09:19:49.517288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.517273    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" containerName="test-curl-7750-14f65c"
Sep 15 09:19:49.517288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.517286    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" containerName="test-curl-7750-14f65c"
Sep 15 09:19:49.517364 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.517342    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" containerName="test-curl-7750-14f65c"
Sep 15 09:19:49.521576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.521554    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:49.526712 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.526679    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7750-b1197e"]
Sep 15 09:19:49.660382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.660287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cljt8\" (UniqueName: \"kubernetes.io/projected/93c1e701-b743-4192-8b3a-378d39fb2527-kube-api-access-cljt8\") pod \"test-curl-7750-b1197e\" (UID: \"93c1e701-b743-4192-8b3a-378d39fb2527\") " pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:49.715520 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:49.715491    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7750-14f65c" container="test-curl-7750-14f65c"
Sep 15 09:19:49.761025 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.760988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljt8\" (UniqueName: \"kubernetes.io/projected/93c1e701-b743-4192-8b3a-378d39fb2527-kube-api-access-cljt8\") pod \"test-curl-7750-b1197e\" (UID: \"93c1e701-b743-4192-8b3a-378d39fb2527\") " pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:49.768354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.768330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljt8\" (UniqueName: \"kubernetes.io/projected/93c1e701-b743-4192-8b3a-378d39fb2527-kube-api-access-cljt8\") pod \"test-curl-7750-b1197e\" (UID: \"93c1e701-b743-4192-8b3a-378d39fb2527\") " pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:49.834336 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:49.834294    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:50.102555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.102197    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7750-b1197e"]
Sep 15 09:19:50.105265 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:50.105237    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93c1e701_b743_4192_8b3a_378d39fb2527.slice/crio-d63d5a39742448d2497b5ab199d3c7c012c979ec60e33d60867714833e58d4d0 WatchSource:0}: Error finding container d63d5a39742448d2497b5ab199d3c7c012c979ec60e33d60867714833e58d4d0: Status 404 returned error can't find the container with id d63d5a39742448d2497b5ab199d3c7c012c979ec60e33d60867714833e58d4d0
Sep 15 09:19:50.348575 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.348536    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"]
Sep 15 09:19:50.352603 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.352539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.356033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.355957    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ptrgq\""
Sep 15 09:19:50.356169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.356119    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 09:19:50.356169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.356125    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 09:19:50.356294 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.356229    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a23d9f-serving-cert\""
Sep 15 09:19:50.356393 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.356358    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 09:19:50.363085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.362953    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"]
Sep 15 09:19:50.391149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.391117    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:50.467060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.467013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89f547-857b-46ba-ae32-7e9f960fb392-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.467060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.467064    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwz8s\" (UniqueName: \"kubernetes.io/projected/ad89f547-857b-46ba-ae32-7e9f960fb392-kube-api-access-bwz8s\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.467306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.467149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.467306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.467181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-metrics-certs\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.490966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.490930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:50.567891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.567856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.567891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.567890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-metrics-certs\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.568396 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.567964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89f547-857b-46ba-ae32-7e9f960fb392-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.568396 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.567985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwz8s\" (UniqueName: \"kubernetes.io/projected/ad89f547-857b-46ba-ae32-7e9f960fb392-kube-api-access-bwz8s\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.568396 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:50.568037    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-serving-cert: secret "maas-api-e2e-inf-a-a23d9f-serving-cert" not found
Sep 15 09:19:50.568396 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:50.568131    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls podName:ad89f547-857b-46ba-ae32-7e9f960fb392 nodeName:}" failed. No retries permitted until 2026-09-15 09:19:51.068111136 +0000 UTC m=+783.527400658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls") pod "maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" (UID: "ad89f547-857b-46ba-ae32-7e9f960fb392") : secret "maas-api-e2e-inf-a-a23d9f-serving-cert" not found
Sep 15 09:19:50.568641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.568620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89f547-857b-46ba-ae32-7e9f960fb392-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.570526 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.570509    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-metrics-certs\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.575768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.575747    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwz8s\" (UniqueName: \"kubernetes.io/projected/ad89f547-857b-46ba-ae32-7e9f960fb392-kube-api-access-bwz8s\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:50.665053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.664969    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:50.723226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.723188    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7750-b1197e" event={"ID":"93c1e701-b743-4192-8b3a-378d39fb2527","Type":"ContainerStarted","Data":"b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f"}
Sep 15 09:19:50.723226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.723239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7750-b1197e" event={"ID":"93c1e701-b743-4192-8b3a-378d39fb2527","Type":"ContainerStarted","Data":"d63d5a39742448d2497b5ab199d3c7c012c979ec60e33d60867714833e58d4d0"}
Sep 15 09:19:50.723463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.723325    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7750-14f65c" podUID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" containerName="test-curl-7750-14f65c" containerID="cri-o://852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a" gracePeriod=2
Sep 15 09:19:50.738995 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.738941    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7750-b1197e" podStartSLOduration=1.738924641 podStartE2EDuration="1.738924641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:50.737100878 +0000 UTC m=+783.196390423" watchObservedRunningTime="2026-09-15 09:19:50.738924641 +0000 UTC m=+783.198214182"
Sep 15 09:19:50.742231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:50.742196    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:51.005114 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.005030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:51.073657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.073609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:51.076031 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.076008    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls\") pod \"maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:51.102330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.102300    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:51.264419 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.264333    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:51.282470 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.282443    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"]
Sep 15 09:19:51.286166 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.286144    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.288933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.288909    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ca71c041-openshift-default-dockercfg-5cgxm\""
Sep 15 09:19:51.299883 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.299852    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"]
Sep 15 09:19:51.327599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.327564    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:51.376476 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376444    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-credential-socket\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376476 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b845e2-8b98-4073-9833-379129891efc-istiod-ca-cert\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376617    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgl5\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-kube-api-access-kfgl5\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-data\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376736    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-istio-token\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376774    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-socket\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376802    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-envoy\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-certs\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.376922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.376884    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b845e2-8b98-4073-9833-379129891efc-istio-podinfo\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.410218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.410157    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:51.429289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.429254    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7750-b1197e"]
Sep 15 09:19:51.432409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.432371    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7750-b1197e"]
Sep 15 09:19:51.478408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgl5\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-kube-api-access-kfgl5\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.478408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-data\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.478661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-istio-token\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.478661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-socket\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.478661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-envoy\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.478661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-certs\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.478661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b845e2-8b98-4073-9833-379129891efc-istio-podinfo\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-credential-socket\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478703    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b845e2-8b98-4073-9833-379129891efc-istiod-ca-cert\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.478902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-data\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479294 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.479242    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-credential-socket\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479294 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.479273    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-socket\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479497 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.479356    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-certs\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.479670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.479646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b845e2-8b98-4073-9833-379129891efc-istiod-ca-cert\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.481587 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.481546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-envoy\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.481769 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.481748    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b845e2-8b98-4073-9833-379129891efc-istio-podinfo\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.491596 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.491575    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-istio-token\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.492234 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.492215    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgl5\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-kube-api-access-kfgl5\") pod \"e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") " pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.550571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.550547    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"]
Sep 15 09:19:51.552698 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:51.552671    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad89f547_857b_46ba_ae32_7e9f960fb392.slice/crio-03f018558752c0f90236049292d5c4482ece9e870da5532fafdaf3c415537659 WatchSource:0}: Error finding container 03f018558752c0f90236049292d5c4482ece9e870da5532fafdaf3c415537659: Status 404 returned error can't find the container with id 03f018558752c0f90236049292d5c4482ece9e870da5532fafdaf3c415537659
Sep 15 09:19:51.606431 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.606395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:51.728944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.728903    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" event={"ID":"ad89f547-857b-46ba-ae32-7e9f960fb392","Type":"ContainerStarted","Data":"03f018558752c0f90236049292d5c4482ece9e870da5532fafdaf3c415537659"}
Sep 15 09:19:51.729118 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:51.729104    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7750-b1197e" container="test-curl-7750-b1197e"
Sep 15 09:19:51.882191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.882166    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"]
Sep 15 09:19:51.884431 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:51.884396    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b845e2_8b98_4073_9833_379129891efc.slice/crio-cd9035698c94098d1ff69451a880778961de4a9f14ebdaff8a20e13f81201fbc WatchSource:0}: Error finding container cd9035698c94098d1ff69451a880778961de4a9f14ebdaff8a20e13f81201fbc: Status 404 returned error can't find the container with id cd9035698c94098d1ff69451a880778961de4a9f14ebdaff8a20e13f81201fbc
Sep 15 09:19:51.886641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.886611    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:51.886703 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.886676    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:51.886740 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:51.886701    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:52.000721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.000678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:52.291978 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.291875    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:52.322847 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.322005    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"]
Sep 15 09:19:52.322847 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.322560    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c1e701-b743-4192-8b3a-378d39fb2527" containerName="test-curl-7750-b1197e"
Sep 15 09:19:52.322847 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.322575    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c1e701-b743-4192-8b3a-378d39fb2527" containerName="test-curl-7750-b1197e"
Sep 15 09:19:52.322847 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.322662    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c1e701-b743-4192-8b3a-378d39fb2527" containerName="test-curl-7750-b1197e"
Sep 15 09:19:52.326605 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.326584    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.331565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.331080    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-abf07f-openshift-default-dockercfg-75zqj\""
Sep 15 09:19:52.348771 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.348700    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"]
Sep 15 09:19:52.391865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.391798    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:52.490411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-credential-socket\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490578 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqcg\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-kube-api-access-zkqcg\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490578 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istiod-ca-cert\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490578 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-token\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490578 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-data\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490628    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-envoy\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-socket\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490731    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-certs\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.490787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.490758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-podinfo\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.527582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.527548    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591656    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-envoy\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-socket\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591800    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-certs\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-podinfo\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-credential-socket\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqcg\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-kube-api-access-zkqcg\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istiod-ca-cert\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.591984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-token\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.592053    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-data\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.592463    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-data\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.592880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.592733    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-credential-socket\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.593104 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.593064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-socket\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.593283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.593223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istiod-ca-cert\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.593283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.593241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-certs\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.596112 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.596087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-envoy\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.596232 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.596179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-podinfo\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.601033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.600983    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqcg\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-kube-api-access-zkqcg\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.601283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.601241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-token\") pod \"e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") " pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.646683 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.646636    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:52.744991 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.743782    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" event={"ID":"c1b845e2-8b98-4073-9833-379129891efc","Type":"ContainerStarted","Data":"103a303fb26146dff5b0e326f5cac422c1ca3bf3ecde4d18f993584e05f56ec1"}
Sep 15 09:19:52.744991 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.743872    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" event={"ID":"c1b845e2-8b98-4073-9833-379129891efc","Type":"ContainerStarted","Data":"cd9035698c94098d1ff69451a880778961de4a9f14ebdaff8a20e13f81201fbc"}
Sep 15 09:19:52.744991 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.744236    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7750-b1197e" podUID="93c1e701-b743-4192-8b3a-378d39fb2527" containerName="test-curl-7750-b1197e" containerID="cri-o://b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f" gracePeriod=2
Sep 15 09:19:52.765052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.764988    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" podStartSLOduration=1.764969222 podStartE2EDuration="1.764969222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:52.763192831 +0000 UTC m=+785.222482372" watchObservedRunningTime="2026-09-15 09:19:52.764969222 +0000 UTC m=+785.224258763"
Sep 15 09:19:52.938856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:52.938810    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"]
Sep 15 09:19:53.017350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.017290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:53.590031 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.590005    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:53.608262 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.606994    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:53.612945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.612914    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:53.702778 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.702738    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgdj4\" (UniqueName: \"kubernetes.io/projected/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1-kube-api-access-xgdj4\") pod \"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1\" (UID: \"5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1\") "
Sep 15 09:19:53.705493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.705463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1-kube-api-access-xgdj4" (OuterVolumeSpecName: "kube-api-access-xgdj4") pod "5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" (UID: "5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1"). InnerVolumeSpecName "kube-api-access-xgdj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:19:53.750779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.750746    2571 generic.go:356] "Generic (PLEG): container finished" podID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" containerID="852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a" exitCode=137
Sep 15 09:19:53.750996 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.750814    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:53.750996 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.750908    2571 scope.go:117] "RemoveContainer" containerID="852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a"
Sep 15 09:19:53.751350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.751331    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:53.753456 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.753417    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:19:53.774141 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:53.774097    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7750-14f65c\" is forbidden: User \"system:node:ip-10-0-134-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-28.ec2.internal' and this object" podUID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" pod="opendatahub/test-curl-7750-14f65c"
Sep 15 09:19:53.805312 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.805241    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgdj4\" (UniqueName: \"kubernetes.io/projected/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1-kube-api-access-xgdj4\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:19:53.866465 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:53.866424    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:54.040693 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:19:54.040650    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a0dccb_1cc9_4c07_8d3f_bdd953c3091e.slice/crio-14dd7c8c3e7d32f386d85bcfcceb0588876c2457f7f2b253182b1633a14bc8a3 WatchSource:0}: Error finding container 14dd7c8c3e7d32f386d85bcfcceb0588876c2457f7f2b253182b1633a14bc8a3: Status 404 returned error can't find the container with id 14dd7c8c3e7d32f386d85bcfcceb0588876c2457f7f2b253182b1633a14bc8a3
Sep 15 09:19:54.043223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.043191    2571 scope.go:117] "RemoveContainer" containerID="852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a"
Sep 15 09:19:54.043338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.043258    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:54.043338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.043328    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:54.043452 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.043359    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:19:54.043564 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:54.043542    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a\": container with ID starting with 852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a not found: ID does not exist" containerID="852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a"
Sep 15 09:19:54.043624 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.043573    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a"} err="failed to get container status \"852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a\": rpc error: code = NotFound desc = could not find container \"852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a\": container with ID starting with 852536668c3634dd83980b72062115e6fdb5ea6c73ad5ed5e2bb2d901957dd5a not found: ID does not exist"
Sep 15 09:19:54.057147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.057114    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1" path="/var/lib/kubelet/pods/5ceed7d4-7e7e-452d-8f6b-c95c1c2277e1/volumes"
Sep 15 09:19:54.328140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.328105    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:54.591397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.591355    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:54.760718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.760684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" event={"ID":"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e","Type":"ContainerStarted","Data":"4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"}
Sep 15 09:19:54.761226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.761189    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" event={"ID":"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e","Type":"ContainerStarted","Data":"14dd7c8c3e7d32f386d85bcfcceb0588876c2457f7f2b253182b1633a14bc8a3"}
Sep 15 09:19:54.763784 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.763756    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" event={"ID":"ad89f547-857b-46ba-ae32-7e9f960fb392","Type":"ContainerStarted","Data":"982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31"}
Sep 15 09:19:54.764009 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.763983    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:19:54.783030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.782979    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podStartSLOduration=2.782965496 podStartE2EDuration="2.782965496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:54.780154865 +0000 UTC m=+787.239444408" watchObservedRunningTime="2026-09-15 09:19:54.782965496 +0000 UTC m=+787.242255036"
Sep 15 09:19:54.799494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.799452    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" podStartSLOduration=2.24790156 podStartE2EDuration="4.799433872s" totalImagesPullingTime="2.551532312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:19:54.796982883 +0000 UTC m=+787.256272425" watchObservedRunningTime="2026-09-15 09:19:54.799433872 +0000 UTC m=+787.258723413"
Sep 15 09:19:54.937513 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.937449    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:54.989550 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:54.989516    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:55.020848 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.019970    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cljt8\" (UniqueName: \"kubernetes.io/projected/93c1e701-b743-4192-8b3a-378d39fb2527-kube-api-access-cljt8\") pod \"93c1e701-b743-4192-8b3a-378d39fb2527\" (UID: \"93c1e701-b743-4192-8b3a-378d39fb2527\") "
Sep 15 09:19:55.023190 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.023152    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c1e701-b743-4192-8b3a-378d39fb2527-kube-api-access-cljt8" (OuterVolumeSpecName: "kube-api-access-cljt8") pod "93c1e701-b743-4192-8b3a-378d39fb2527" (UID: "93c1e701-b743-4192-8b3a-378d39fb2527"). InnerVolumeSpecName "kube-api-access-cljt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:19:55.122153 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.122051    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cljt8\" (UniqueName: \"kubernetes.io/projected/93c1e701-b743-4192-8b3a-378d39fb2527-kube-api-access-cljt8\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:19:55.647077 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.647028    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:55.652526 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.652490    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:55.771766 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.771731    2571 generic.go:356] "Generic (PLEG): container finished" podID="93c1e701-b743-4192-8b3a-378d39fb2527" containerID="b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f" exitCode=137
Sep 15 09:19:55.772227 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.771780    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:55.772227 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.771822    2571 scope.go:117] "RemoveContainer" containerID="b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f"
Sep 15 09:19:55.773398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.773376    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:55.774610 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.774584    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:19:55.782225 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.782206    2571 scope.go:117] "RemoveContainer" containerID="b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f"
Sep 15 09:19:55.782514 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:55.782494    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f\": container with ID starting with b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f not found: ID does not exist" containerID="b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f"
Sep 15 09:19:55.782582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:55.782528    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f"} err="failed to get container status \"b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f\": rpc error: code = NotFound desc = could not find container \"b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f\": container with ID starting with b03afe04e2cd74fb5876e6ae351155768cbd7d37010a37911da5be9708b3e17f not found: ID does not exist"
Sep 15 09:19:55.797855 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:55.797811    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7750-b1197e\" is forbidden: User \"system:node:ip-10-0-134-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-28.ec2.internal' and this object" podUID="93c1e701-b743-4192-8b3a-378d39fb2527" pod="opendatahub/test-curl-7750-b1197e"
Sep 15 09:19:56.055629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:56.055549    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c1e701-b743-4192-8b3a-378d39fb2527" path="/var/lib/kubelet/pods/93c1e701-b743-4192-8b3a-378d39fb2527/volumes"
Sep 15 09:19:56.176661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:56.176614    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:56.844731 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:56.844696    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:56.943185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:56.943147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:57.453614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:57.453578    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:58.416599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:58.416561    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:58.476500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:58.476465    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:58.872570 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:58.872505    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:58.975408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:58.975375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:59.073582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.073553    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:59.168862 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.168766    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:59.550910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.550812    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"]
Sep 15 09:19:59.555517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.555499    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.557775 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.557752    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-abf07f-serving-cert\""
Sep 15 09:19:59.564084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.564054    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"]
Sep 15 09:19:59.640851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.633564    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:59.668247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.668214    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.668403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.668269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-metrics-certs\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.668403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.668319    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-openshift-service-ca\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.668403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.668356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmcz\" (UniqueName: \"kubernetes.io/projected/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-kube-api-access-bcmcz\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.720732 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.720688    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:19:59.769376 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.769339    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-metrics-certs\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.769567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.769385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-openshift-service-ca\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.769567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.769423    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmcz\" (UniqueName: \"kubernetes.io/projected/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-kube-api-access-bcmcz\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.769567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.769479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.769807 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:59.769579    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-serving-cert: secret "maas-api-e2e-autores-abf07f-serving-cert" not found
Sep 15 09:19:59.769807 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:19:59.769636    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls podName:21f24a65-0c3a-41f5-83b6-9b9c0b1c7841 nodeName:}" failed. No retries permitted until 2026-09-15 09:20:00.269619985 +0000 UTC m=+792.728909506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls") pod "maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" (UID: "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841") : secret "maas-api-e2e-autores-abf07f-serving-cert" not found
Sep 15 09:19:59.770153 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.770133    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-openshift-service-ca\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.771778 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.771759    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-metrics-certs\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:19:59.777662 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:19:59.777641    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmcz\" (UniqueName: \"kubernetes.io/projected/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-kube-api-access-bcmcz\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:00.273333 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:00.273297    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:00.273514 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:20:00.273468    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-serving-cert: secret "maas-api-e2e-autores-abf07f-serving-cert" not found
Sep 15 09:20:00.273576 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:20:00.273550    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls podName:21f24a65-0c3a-41f5-83b6-9b9c0b1c7841 nodeName:}" failed. No retries permitted until 2026-09-15 09:20:01.273528167 +0000 UTC m=+793.732817704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls") pod "maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" (UID: "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841") : secret "maas-api-e2e-autores-abf07f-serving-cert" not found
Sep 15 09:20:00.299978 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:00.299946    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:00.400803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:00.400771    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:00.778861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:00.778817    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:20:00.911991 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:00.911907    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:01.011664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.011630    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:01.285357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.285255    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:01.287779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.287752    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") pod \"maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:01.367556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.367507    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:01.657079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.657048    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"]
Sep 15 09:20:01.659305 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:20:01.659203    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f24a65_0c3a_41f5_83b6_9b9c0b1c7841.slice/crio-e262c3a69c2f0f4d4a849520f54a1e0104d78ba58c204ef7cc5435ea9797a408 WatchSource:0}: Error finding container e262c3a69c2f0f4d4a849520f54a1e0104d78ba58c204ef7cc5435ea9797a408: Status 404 returned error can't find the container with id e262c3a69c2f0f4d4a849520f54a1e0104d78ba58c204ef7cc5435ea9797a408
Sep 15 09:20:01.676646 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.676617    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:01.778423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.778384    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:01.811690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.811650    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" event={"ID":"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841","Type":"ContainerStarted","Data":"e262c3a69c2f0f4d4a849520f54a1e0104d78ba58c204ef7cc5435ea9797a408"}
Sep 15 09:20:01.883926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.883892    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:01.977344 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:01.977262    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:02.085127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:02.085092    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:02.818228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:02.818191    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" event={"ID":"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841","Type":"ContainerStarted","Data":"62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829"}
Sep 15 09:20:02.818614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:02.818312    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:02.841840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:02.841765    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" podStartSLOduration=3.300499298 podStartE2EDuration="3.841740877s" totalImagesPullingTime="541.241579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:02.83940173 +0000 UTC m=+795.298691272" watchObservedRunningTime="2026-09-15 09:20:02.841740877 +0000 UTC m=+795.301030421"
Sep 15 09:20:03.120607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.120578    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:03.410896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.405371    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:20:03.824632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.824518    2571 generic.go:356] "Generic (PLEG): container finished" podID="25538817-dc63-42ee-a527-35d8b90c0207" containerID="ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4" exitCode=0
Sep 15 09:20:03.824632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.824593    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" event={"ID":"25538817-dc63-42ee-a527-35d8b90c0207","Type":"ContainerDied","Data":"ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"}
Sep 15 09:20:03.825140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.825120    2571 scope.go:117] "RemoveContainer" containerID="ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"
Sep 15 09:20:03.826390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.826362    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:03.826465 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.826437    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:03.826504 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:03.826469    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:04.383236 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:04.383200    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:04.835318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:04.835270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" event={"ID":"25538817-dc63-42ee-a527-35d8b90c0207","Type":"ContainerStarted","Data":"2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6"}
Sep 15 09:20:04.840315 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:04.840172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:04.985523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:04.985474    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:05.053422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:05.053377    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:20:05.053668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:05.053630    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 15 09:20:05.053807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:05.053686    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 15 09:20:05.769113 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:05.769071    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:05.873922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:05.873884    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:05.933271 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:05.933229    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:06.054926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:06.054387    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 15 09:20:06.054926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:06.054451    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 15 09:20:06.299284 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:06.299246    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:06.400874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:06.400812    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:06.500900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:06.500864    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:07.053224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:07.053172    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 15 09:20:07.053742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:07.053251    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 15 09:20:07.233121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:07.233024    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:07.713506 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:07.713469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:08.053189 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:08.053078    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 15 09:20:08.053189 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:08.053153    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 15 09:20:08.362697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:08.362665    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:08.839354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:08.839315    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:20:09.053903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:09.053855    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 15 09:20:09.054404 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:09.053929    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 15 09:20:09.426519 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:09.426485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:10.053010 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.052970    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 15 09:20:10.053217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.053043    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 15 09:20:10.216538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.216455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:10.341675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.341639    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:10.502405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.502376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:10.609498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.609388    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:10.941447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:10.941374    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:11.057796 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.057764    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:20:11.058080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.058055    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:20:11.058871 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.058851    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:20:11.297817 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.297724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:11.573526 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.573430    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"]
Sep 15 09:20:11.581044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.581015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.583573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.583549    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2430e2-openshift-default-dockercfg-r48bm\""
Sep 15 09:20:11.590095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.590064    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"]
Sep 15 09:20:11.598107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.598077    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:11.654540 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.654503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:11.712566 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-certs\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-data\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712637    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-credential-socket\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-istio-token\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hl57\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-kube-api-access-7hl57\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045d9d4-268e-4f08-a808-54faf133fae4-istio-podinfo\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712851    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-envoy\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-socket\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.712946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.712938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045d9d4-268e-4f08-a808-54faf133fae4-istiod-ca-cert\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814633 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-socket\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814633 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045d9d4-268e-4f08-a808-54faf133fae4-istiod-ca-cert\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-certs\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-data\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-credential-socket\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-istio-token\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.814919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814894    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hl57\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-kube-api-access-7hl57\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814924    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045d9d4-268e-4f08-a808-54faf133fae4-istio-podinfo\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.814966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-envoy\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.815152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-certs\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.815158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-socket\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.815353    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-data\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815537 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.815506    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045d9d4-268e-4f08-a808-54faf133fae4-istiod-ca-cert\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.815680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.815643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-credential-socket\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.817591 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.817563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-envoy\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.817953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.817932    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045d9d4-268e-4f08-a808-54faf133fae4-istio-podinfo\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.826065 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.826000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-istio-token\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.826431 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.826408    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hl57\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-kube-api-access-7hl57\") pod \"e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") " pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.893625 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.893587    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:11.969366 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:11.968694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:12.180811 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.180779    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"]
Sep 15 09:20:12.184571 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:20:12.184537    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2045d9d4_268e_4f08_a808_54faf133fae4.slice/crio-5c9171ad9a41a7512482ff6f9ca7af59af7008e173c071d9e09f0d6695a56d6a WatchSource:0}: Error finding container 5c9171ad9a41a7512482ff6f9ca7af59af7008e173c071d9e09f0d6695a56d6a: Status 404 returned error can't find the container with id 5c9171ad9a41a7512482ff6f9ca7af59af7008e173c071d9e09f0d6695a56d6a
Sep 15 09:20:12.187303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.187263    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:12.187390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.187339    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:12.187390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.187374    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:12.608102 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.608068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:12.893529 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.893435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" event={"ID":"2045d9d4-268e-4f08-a808-54faf133fae4","Type":"ContainerStarted","Data":"8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"}
Sep 15 09:20:12.893529 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.893502    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" event={"ID":"2045d9d4-268e-4f08-a808-54faf133fae4","Type":"ContainerStarted","Data":"5c9171ad9a41a7512482ff6f9ca7af59af7008e173c071d9e09f0d6695a56d6a"}
Sep 15 09:20:12.893746 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.893707    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:12.914069 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:12.914014    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podStartSLOduration=1.913995791 podStartE2EDuration="1.913995791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:12.912759015 +0000 UTC m=+805.372048582" watchObservedRunningTime="2026-09-15 09:20:12.913995791 +0000 UTC m=+805.373285332"
Sep 15 09:20:13.585430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:13.585394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:13.899938 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:13.899854    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:14.572691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.572646    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:14.799015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.798887    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:14.888210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.888158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:14.904813 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.904783    2571 generic.go:356] "Generic (PLEG): container finished" podID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerID="4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184" exitCode=0
Sep 15 09:20:14.905180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.904856    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" event={"ID":"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e","Type":"ContainerDied","Data":"4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"}
Sep 15 09:20:14.905222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.905197    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:14.905387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.905372    2571 scope.go:117] "RemoveContainer" containerID="4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"
Sep 15 09:20:14.906518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.906489    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:14.906623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.906556    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:14.906623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.906587    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:14.906710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:14.906662    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:15.460306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:15.460279    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:15.663328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:15.663284    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:15.912021 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:15.911981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" event={"ID":"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e","Type":"ContainerStarted","Data":"35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3"}
Sep 15 09:20:16.202674 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:16.202600    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:16.628238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:16.628197    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:16.648050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:16.647544    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:20:16.648050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:16.647681    2571 patch_prober.go:28] interesting pod/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 15 09:20:16.648050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:16.647747    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 15 09:20:17.029474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.029375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:17.176143 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.176104    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:17.323233 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.323128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:17.418739 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.418701    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:17.520002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.519909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:17.648055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.647511    2571 patch_prober.go:28] interesting pod/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 15 09:20:17.648055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:17.647599    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 15 09:20:18.047511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.047417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:18.149151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.149121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:18.451397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.451308    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:18.548549 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.548460    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:18.627523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.627405    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:18.649160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.648675    2571 patch_prober.go:28] interesting pod/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 15 09:20:18.649160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.648741    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 15 09:20:18.878149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:18.878103    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:19.603998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:19.603958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:19.647484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:19.647450    2571 patch_prober.go:28] interesting pod/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 15 09:20:19.647665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:19.647512    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 15 09:20:20.401860 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:20.401810    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:20.647562 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:20.647520    2571 patch_prober.go:28] interesting pod/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 15 09:20:20.648062 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:20.647592    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 15 09:20:21.096880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.096850    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:21.651787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.651756    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:20:21.652287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.652266    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:20:21.653019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.652993    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:20:21.856215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.856182    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:21.955910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.955800    2571 generic.go:356] "Generic (PLEG): container finished" podID="c1b845e2-8b98-4073-9833-379129891efc" containerID="103a303fb26146dff5b0e326f5cac422c1ca3bf3ecde4d18f993584e05f56ec1" exitCode=0
Sep 15 09:20:21.955910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.955873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" event={"ID":"c1b845e2-8b98-4073-9833-379129891efc","Type":"ContainerDied","Data":"103a303fb26146dff5b0e326f5cac422c1ca3bf3ecde4d18f993584e05f56ec1"}
Sep 15 09:20:21.956327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.956312    2571 scope.go:117] "RemoveContainer" containerID="103a303fb26146dff5b0e326f5cac422c1ca3bf3ecde4d18f993584e05f56ec1"
Sep 15 09:20:21.957484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.957458    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:21.957581 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.957539    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:21.957645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:21.957590    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:22.646600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:22.646565    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:22.910873 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:22.910765    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:20:22.964360 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:22.964311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" event={"ID":"c1b845e2-8b98-4073-9833-379129891efc","Type":"ContainerStarted","Data":"d274de55b6f641b63324710c5d9c65b095527e4d6de25c12b21ddc250f6428db"}
Sep 15 09:20:23.607380 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:23.607337    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:20:23.608488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:23.608460    2571 patch_prober.go:28] interesting pod/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 09:20:23.608653 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:23.608544    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 09:20:23.751437 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:23.751392    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:20:23.827762 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:23.827699    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:23.928727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:23.928632    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:24.607725 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:24.607693    2571 patch_prober.go:28] interesting pod/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 09:20:24.608029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:24.608005    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 09:20:24.668908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:24.668863    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:25.357461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.357364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:25.430778 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.430685    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:25.529528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.529469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:25.607269 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.607219    2571 patch_prober.go:28] interesting pod/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 15 09:20:25.607517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.607280    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 15 09:20:25.626704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.626670    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:25.689853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.679594    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"]
Sep 15 09:20:25.701098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.701067    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:25.982112 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:25.982019    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy" containerID="cri-o://d274de55b6f641b63324710c5d9c65b095527e4d6de25c12b21ddc250f6428db" gracePeriod=30
Sep 15 09:20:26.035751 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.035588    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:26.173736 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.173700    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:26.272084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.271993    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:26.429260 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.429206    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:26.663218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.663172    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"]
Sep 15 09:20:26.667647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.667617    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.673864 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.672331    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-0cec7519-gw-openshift-default-dockercfg-fsgtz\""
Sep 15 09:20:26.684479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.684440    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"]
Sep 15 09:20:26.708955 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.708847    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:26.800614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-podinfo\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.800614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-credential-socket\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.800892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-certs\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.800892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800685    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-envoy\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.800892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-socket\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.800892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmmx\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-kube-api-access-hhmmx\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.800892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800810    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-data\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.801180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800912    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a5f141-951d-47ba-bf30-6d8a2be97fed-istiod-ca-cert\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.801180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.800994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-token\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902275 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-token\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-podinfo\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902308    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-credential-socket\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-certs\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-envoy\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-socket\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmmx\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-kube-api-access-hhmmx\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-data\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902718 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a5f141-951d-47ba-bf30-6d8a2be97fed-istiod-ca-cert\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.902941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-socket\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.903007 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.902940    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-data\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.903170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.903148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-credential-socket\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.903222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.903209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-certs\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.903299 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.903280    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a5f141-951d-47ba-bf30-6d8a2be97fed-istiod-ca-cert\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.905163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.905142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-envoy\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.905340 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.905322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-podinfo\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.923901 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.921255    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmmx\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-kube-api-access-hhmmx\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.923901 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.923296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-token\") pod \"e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") " pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:26.948703 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.948664    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:26.985642 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:26.985545    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:27.049592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.049468    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:27.316993 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.316869    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"]
Sep 15 09:20:27.327339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.326122    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:27.327339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.326209    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:27.327339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.326252    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:27.768510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.768435    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:27.993519 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.993477    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" event={"ID":"60a5f141-951d-47ba-bf30-6d8a2be97fed","Type":"ContainerStarted","Data":"58d5a6d402533ea774fd196149aa4ea35bdf9c92c2fa9a8824ab421eb40c4769"}
Sep 15 09:20:27.993736 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:27.993531    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" event={"ID":"60a5f141-951d-47ba-bf30-6d8a2be97fed","Type":"ContainerStarted","Data":"bfc8085d4fc63cd7cb401b3d870a8ca0ee647dc607dc5706ca561ff30989c169"}
Sep 15 09:20:28.029918 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:28.024463    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podStartSLOduration=2.0244335 podStartE2EDuration="2.0244335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:28.014541635 +0000 UTC m=+820.473831177" watchObservedRunningTime="2026-09-15 09:20:28.0244335 +0000 UTC m=+820.483723042"
Sep 15 09:20:28.654100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:28.654060    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:28.986896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:28.986745    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:28.992790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:28.992753    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:28.999213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:28.999177    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:29.000421 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:29.000394    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:29.450032 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:29.449994    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:29.550547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:29.550514    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:29.654541 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:29.654507    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:30.545658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:30.545619    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:30.947019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:30.946981    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 09:20:30.947193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:30.947043    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 09:20:31.403177 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:31.403108    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:31.508580 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:31.508546    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:31.909012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:31.908978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:32.009225 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.009192    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:32.537892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.537853    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:32.890952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.890915    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"]
Sep 15 09:20:32.894386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.894367    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:32.896803 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.896778    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ffb593\"/\"openshift-service-ca.crt\""
Sep 15 09:20:32.896931 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.896883    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffb593\"/\"default-dockercfg-bjtj2\""
Sep 15 09:20:32.897903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.897886    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ffb593\"/\"kube-root-ca.crt\""
Sep 15 09:20:32.898044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.898025    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffb593\"/\"simulator-w0-ffb593-kserve-self-signed-certs\""
Sep 15 09:20:32.903981 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.903955    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"]
Sep 15 09:20:32.974973 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.974936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-dshm\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:32.974973 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.974977    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-tmp-dir\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:32.975389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.975011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc38895-56fe-41cb-b5d8-5df27fe97798-tls-certs\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:32.975389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.975040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtvrl\" (UniqueName: \"kubernetes.io/projected/cdc38895-56fe-41cb-b5d8-5df27fe97798-kube-api-access-wtvrl\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:32.975389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.975107    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-model-cache\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:32.975389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:32.975146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-home\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.022475 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.022442    2571 generic.go:356] "Generic (PLEG): container finished" podID="2045d9d4-268e-4f08-a808-54faf133fae4" containerID="8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf" exitCode=0
Sep 15 09:20:33.022626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.022515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" event={"ID":"2045d9d4-268e-4f08-a808-54faf133fae4","Type":"ContainerDied","Data":"8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"}
Sep 15 09:20:33.022982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.022966    2571 scope.go:117] "RemoveContainer" containerID="8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"
Sep 15 09:20:33.024235 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.024207    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:33.024311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.024274    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:33.024311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.024306    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:33.075893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.075859    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-dshm\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.075893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.075895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-tmp-dir\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.076147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.075938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc38895-56fe-41cb-b5d8-5df27fe97798-tls-certs\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.076147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.076021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvrl\" (UniqueName: \"kubernetes.io/projected/cdc38895-56fe-41cb-b5d8-5df27fe97798-kube-api-access-wtvrl\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.076902 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.076606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-model-cache\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.076902 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.076706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-home\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.076902 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.076868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-tmp-dir\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.077148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.076991    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-model-cache\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.077148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.077074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-home\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.079045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.079013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-dshm\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.079388 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.079368    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc38895-56fe-41cb-b5d8-5df27fe97798-tls-certs\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.091193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.091146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvrl\" (UniqueName: \"kubernetes.io/projected/cdc38895-56fe-41cb-b5d8-5df27fe97798-kube-api-access-wtvrl\") pod \"simulator-w0-ffb593-kserve-5f765d8494-5zs57\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") " pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.108305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.108271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:33.205497 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.205398    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:33.305518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.305483    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:33.493122 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:33.493088    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"]
Sep 15 09:20:33.494770 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:20:33.494720    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc38895_56fe_41cb_b5d8_5df27fe97798.slice/crio-265a42aa38a8dd56e8e9bda6b5f7a7fe4a781f953c9e16d139c01ab417052b57 WatchSource:0}: Error finding container 265a42aa38a8dd56e8e9bda6b5f7a7fe4a781f953c9e16d139c01ab417052b57: Status 404 returned error can't find the container with id 265a42aa38a8dd56e8e9bda6b5f7a7fe4a781f953c9e16d139c01ab417052b57
Sep 15 09:20:34.015033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.015002    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:34.042853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.035703    2571 generic.go:356] "Generic (PLEG): container finished" podID="c1b845e2-8b98-4073-9833-379129891efc" containerID="d274de55b6f641b63324710c5d9c65b095527e4d6de25c12b21ddc250f6428db" exitCode=0
Sep 15 09:20:34.042853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.035795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" event={"ID":"c1b845e2-8b98-4073-9833-379129891efc","Type":"ContainerDied","Data":"d274de55b6f641b63324710c5d9c65b095527e4d6de25c12b21ddc250f6428db"}
Sep 15 09:20:34.042853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.035868    2571 scope.go:117] "RemoveContainer" containerID="103a303fb26146dff5b0e326f5cac422c1ca3bf3ecde4d18f993584e05f56ec1"
Sep 15 09:20:34.043488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.042816    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" event={"ID":"2045d9d4-268e-4f08-a808-54faf133fae4","Type":"ContainerStarted","Data":"72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586"}
Sep 15 09:20:34.052478 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.046292    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57" event={"ID":"cdc38895-56fe-41cb-b5d8-5df27fe97798","Type":"ContainerStarted","Data":"c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734"}
Sep 15 09:20:34.052478 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.046347    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57" event={"ID":"cdc38895-56fe-41cb-b5d8-5df27fe97798","Type":"ContainerStarted","Data":"265a42aa38a8dd56e8e9bda6b5f7a7fe4a781f953c9e16d139c01ab417052b57"}
Sep 15 09:20:34.080199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.079513    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:20:34.087009 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.086974    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b845e2-8b98-4073-9833-379129891efc-istio-podinfo\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087032    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b845e2-8b98-4073-9833-379129891efc-istiod-ca-cert\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087070    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-socket\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087155    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgl5\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-kube-api-access-kfgl5\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087301 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087184    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-envoy\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087301 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087211    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-data\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087301 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087275    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-certs\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087327    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-istio-token\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.087394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087382    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-credential-socket\") pod \"c1b845e2-8b98-4073-9833-379129891efc\" (UID: \"c1b845e2-8b98-4073-9833-379129891efc\") "
Sep 15 09:20:34.088068 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.087882    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:20:34.088919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.088508    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:20:34.088919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.088530    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b845e2-8b98-4073-9833-379129891efc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:20:34.090268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.088947    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:20:34.090441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.090403    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:20:34.090689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.090663    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:20:34.090816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.090728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1b845e2-8b98-4073-9833-379129891efc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:20:34.091664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.091629    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:20:34.091745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.091704    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-kube-api-access-kfgl5" (OuterVolumeSpecName: "kube-api-access-kfgl5") pod "c1b845e2-8b98-4073-9833-379129891efc" (UID: "c1b845e2-8b98-4073-9833-379129891efc"). InnerVolumeSpecName "kube-api-access-kfgl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:20:34.109349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.109290    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57" podStartSLOduration=2.109266524 podStartE2EDuration="2.109266524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:34.084785708 +0000 UTC m=+826.544075250" watchObservedRunningTime="2026-09-15 09:20:34.109266524 +0000 UTC m=+826.568556066"
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188642    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188678    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgl5\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-kube-api-access-kfgl5\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188691    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188699    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188708    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188715    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b845e2-8b98-4073-9833-379129891efc-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188726    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b845e2-8b98-4073-9833-379129891efc-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188738    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b845e2-8b98-4073-9833-379129891efc-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.188749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.188752    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b845e2-8b98-4073-9833-379129891efc-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:20:34.711588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.711553    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:34.886532 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.886492    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:34.894091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.894063    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:34.894315 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.894289    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 15 09:20:34.894362 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:34.894341    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 15 09:20:35.052717 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.052635    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"
Sep 15 09:20:35.052717 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.052649    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2" event={"ID":"c1b845e2-8b98-4073-9833-379129891efc","Type":"ContainerDied","Data":"cd9035698c94098d1ff69451a880778961de4a9f14ebdaff8a20e13f81201fbc"}
Sep 15 09:20:35.053240 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.052721    2571 scope.go:117] "RemoveContainer" containerID="d274de55b6f641b63324710c5d9c65b095527e4d6de25c12b21ddc250f6428db"
Sep 15 09:20:35.074485 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.074446    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"]
Sep 15 09:20:35.078451 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.078423    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ca71c041-openshift-default-54b64bbcbd-76wt2"]
Sep 15 09:20:35.591086 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.591050    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:35.894615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.894526    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 15 09:20:35.894615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:35.894585    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 15 09:20:36.055845 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:36.055796    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b845e2-8b98-4073-9833-379129891efc" path="/var/lib/kubelet/pods/c1b845e2-8b98-4073-9833-379129891efc/volumes"
Sep 15 09:20:36.374506 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:36.374472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:36.894161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:36.894124    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 15 09:20:36.894345 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:36.894218    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 15 09:20:37.084499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:37.084466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:37.894522 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:37.894478    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 15 09:20:37.894690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:37.894553    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 15 09:20:37.913548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:37.913511    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:20:38.861812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:38.861784    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:38.894000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:38.893962    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Sep 15 09:20:38.894176 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:38.894020    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 15 09:20:39.184500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:39.184397    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:39.898443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:39.898412    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:39.898904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:39.898702    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:39.899353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:39.899339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:20:43.206182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:43.206141    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:43.206607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:43.206206    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:43.218769 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:43.218739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:44.116451 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:44.116421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:20:45.701776 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.701739    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"]
Sep 15 09:20:45.702354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.702333    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy"
Sep 15 09:20:45.702507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.702495    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy"
Sep 15 09:20:45.702566 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.702560    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy"
Sep 15 09:20:45.702615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.702609    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy"
Sep 15 09:20:45.702795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.702780    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy"
Sep 15 09:20:45.702906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.702873    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b845e2-8b98-4073-9833-379129891efc" containerName="istio-proxy"
Sep 15 09:20:45.705903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.705883    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.709770 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.709670    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"kube-root-ca.crt\""
Sep 15 09:20:45.709770 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.709748    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"default-dockercfg-mq8ln\""
Sep 15 09:20:45.709979 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.709749    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"simulator-w4-890f3e-kserve-self-signed-certs\""
Sep 15 09:20:45.709979 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.709797    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"openshift-service-ca.crt\""
Sep 15 09:20:45.743850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.741999    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"]
Sep 15 09:20:45.837851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.836909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:45.847659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.847613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2bb914-6216-4da1-aab8-6ae238e7116a-tls-certs\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.847659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.847656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-model-cache\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.847908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.847679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-home\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.847908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.847804    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-dshm\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.847908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.847858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-tmp-dir\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.847908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.847887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8br\" (UniqueName: \"kubernetes.io/projected/3e2bb914-6216-4da1-aab8-6ae238e7116a-kube-api-access-jg8br\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2bb914-6216-4da1-aab8-6ae238e7116a-tls-certs\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-model-cache\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-home\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-dshm\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-tmp-dir\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949609    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8br\" (UniqueName: \"kubernetes.io/projected/3e2bb914-6216-4da1-aab8-6ae238e7116a-kube-api-access-jg8br\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949780    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-model-cache\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.949841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.949808    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-home\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.950046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.950027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-tmp-dir\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.951772 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.951742    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-dshm\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.952051 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.952010    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2bb914-6216-4da1-aab8-6ae238e7116a-tls-certs\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:45.959249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:45.959229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8br\" (UniqueName: \"kubernetes.io/projected/3e2bb914-6216-4da1-aab8-6ae238e7116a-kube-api-access-jg8br\") pod \"simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") " pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:46.016262 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:46.016216    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:46.316801 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:46.316767    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"]
Sep 15 09:20:47.123259 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:47.123215    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf" event={"ID":"3e2bb914-6216-4da1-aab8-6ae238e7116a","Type":"ContainerStarted","Data":"96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095"}
Sep 15 09:20:47.123638 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:47.123270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf" event={"ID":"3e2bb914-6216-4da1-aab8-6ae238e7116a","Type":"ContainerStarted","Data":"d7ffe601b652524ca09d68c4327f4cf5617ee6e700571435fbae338e8c4b0728"}
Sep 15 09:20:47.144145 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:47.144088    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf" podStartSLOduration=2.144070277 podStartE2EDuration="2.144070277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:20:47.142188518 +0000 UTC m=+839.601478060" watchObservedRunningTime="2026-09-15 09:20:47.144070277 +0000 UTC m=+839.603359818"
Sep 15 09:20:50.140895 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:50.140863    2571 generic.go:356] "Generic (PLEG): container finished" podID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerID="58d5a6d402533ea774fd196149aa4ea35bdf9c92c2fa9a8824ab421eb40c4769" exitCode=0
Sep 15 09:20:50.141303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:50.140937    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" event={"ID":"60a5f141-951d-47ba-bf30-6d8a2be97fed","Type":"ContainerDied","Data":"58d5a6d402533ea774fd196149aa4ea35bdf9c92c2fa9a8824ab421eb40c4769"}
Sep 15 09:20:50.141436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:50.141419    2571 scope.go:117] "RemoveContainer" containerID="58d5a6d402533ea774fd196149aa4ea35bdf9c92c2fa9a8824ab421eb40c4769"
Sep 15 09:20:50.149101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:50.143055    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:50.149101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:50.143240    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:50.149101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:50.143334    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:20:51.149805 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:51.149770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" event={"ID":"60a5f141-951d-47ba-bf30-6d8a2be97fed","Type":"ContainerStarted","Data":"bb90bbc54f5b134302794c77062ba627cfca605adaf440d9c6b3ae41603999e2"}
Sep 15 09:20:51.986585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:51.986541    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:51.986768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:51.986717    2571 patch_prober.go:28] interesting pod/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 15 09:20:51.986820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:51.986768    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 15 09:20:52.487695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:52.487661    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:52.986400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:52.986361    2571 patch_prober.go:28] interesting pod/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 15 09:20:52.986584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:52.986417    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 15 09:20:53.768094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:53.768054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:53.986583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:53.986542    2571 patch_prober.go:28] interesting pod/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 15 09:20:53.986758 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:53.986606    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 15 09:20:54.441330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:54.441278    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:54.986648 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:54.986612    2571 patch_prober.go:28] interesting pod/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 15 09:20:54.987089 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:54.986668    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 15 09:20:55.281335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:55.281258    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:55.505423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:55.505384    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:20:55.986489 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:55.986441    2571 patch_prober.go:28] interesting pod/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 15 09:20:55.986695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:55.986515    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 15 09:20:56.019703 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.019660    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:56.019924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.019731    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:56.075780 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.075743    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:56.185637 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.185600    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:20:56.269735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.269632    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:56.990893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.990850    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:56.991377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.991139    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:56.991960 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:56.991940    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:20:57.062753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:57.062721    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:57.874380 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:57.874351    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:58.538304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:58.538270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:20:59.375169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:20:59.375130    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:00.070892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:00.070863    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:00.664466 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:00.664435    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:00.915502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:00.915386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:00.983427 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:00.983378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:01.005114 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:01.005080    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"]
Sep 15 09:21:01.005355 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:01.005325    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy" containerID="cri-o://bb90bbc54f5b134302794c77062ba627cfca605adaf440d9c6b3ae41603999e2" gracePeriod=30
Sep 15 09:21:01.044012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:01.043976    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:01.301096 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:01.301002    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:01.479212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:01.479180    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:01.804708 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:01.804670    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:02.138534 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:02.138501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:02.237521 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:02.237485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:03.085903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:03.085868    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:03.981048 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:03.981007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:04.592224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:04.592184    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:04.689193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:04.689158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:04.825585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:04.825554    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:05.881000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.880967    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:05.895721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.895687    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"]
Sep 15 09:21:05.900698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.900677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:05.904424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.904395    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-abf07f\"/\"openshift-service-ca.crt\""
Sep 15 09:21:05.904572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.904428    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-abf07f\"/\"autores-model-abf07f-kserve-self-signed-certs\""
Sep 15 09:21:05.904572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.904441    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-abf07f\"/\"kube-root-ca.crt\""
Sep 15 09:21:05.904572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.904432    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-abf07f\"/\"default-dockercfg-znbbb\""
Sep 15 09:21:05.908817 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.908791    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"]
Sep 15 09:21:05.959850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.959800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-home\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:05.960039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.959867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215e84d6-da28-47f4-83a4-1b9c61931183-tls-certs\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:05.960039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.959888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-tmp-dir\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:05.960039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.959933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-model-cache\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:05.960039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.959960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggj5q\" (UniqueName: \"kubernetes.io/projected/215e84d6-da28-47f4-83a4-1b9c61931183-kube-api-access-ggj5q\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:05.960039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:05.960000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-dshm\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.060721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.060680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-tmp-dir\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.060935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.060733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-model-cache\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.060935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.060758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggj5q\" (UniqueName: \"kubernetes.io/projected/215e84d6-da28-47f4-83a4-1b9c61931183-kube-api-access-ggj5q\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.060935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.060799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-dshm\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.060935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.060874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-home\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.060935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.060906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215e84d6-da28-47f4-83a4-1b9c61931183-tls-certs\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.061223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.061178    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-tmp-dir\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.061223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.061205    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-model-cache\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.061298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.061210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-home\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.063165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.063139    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-dshm\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.063351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.063333    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215e84d6-da28-47f4-83a4-1b9c61931183-tls-certs\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.068189 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.068167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggj5q\" (UniqueName: \"kubernetes.io/projected/215e84d6-da28-47f4-83a4-1b9c61931183-kube-api-access-ggj5q\") pod \"autores-model-abf07f-kserve-6559cc9867-nd9ck\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") " pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.190895 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.190783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:06.211881 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.211817    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:06.229576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.229539    2571 generic.go:356] "Generic (PLEG): container finished" podID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerID="bb90bbc54f5b134302794c77062ba627cfca605adaf440d9c6b3ae41603999e2" exitCode=0
Sep 15 09:21:06.229738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.229614    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" event={"ID":"60a5f141-951d-47ba-bf30-6d8a2be97fed","Type":"ContainerDied","Data":"bb90bbc54f5b134302794c77062ba627cfca605adaf440d9c6b3ae41603999e2"}
Sep 15 09:21:06.229738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.229696    2571 scope.go:117] "RemoveContainer" containerID="58d5a6d402533ea774fd196149aa4ea35bdf9c92c2fa9a8824ab421eb40c4769"
Sep 15 09:21:06.296983 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.296784    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:21:06.363144 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363103    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-credential-socket\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363144 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363149    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-podinfo\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363236    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-envoy\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363262    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-socket\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363316    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-data\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363342    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-certs\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363387    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-token\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363655 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363416    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a5f141-951d-47ba-bf30-6d8a2be97fed-istiod-ca-cert\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363655 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363440    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:21:06.363655 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363452    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhmmx\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-kube-api-access-hhmmx\") pod \"60a5f141-951d-47ba-bf30-6d8a2be97fed\" (UID: \"60a5f141-951d-47ba-bf30-6d8a2be97fed\") "
Sep 15 09:21:06.363818 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363657    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:21:06.363930 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363910    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:21:06.364002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.363970    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-data" (OuterVolumeSpecName: "istio-data") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:21:06.364063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.364008    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a5f141-951d-47ba-bf30-6d8a2be97fed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:21:06.364063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.364033    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.364063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.364051    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.364063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.364060    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.365663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.365634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:21:06.365767 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.365671    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-kube-api-access-hhmmx" (OuterVolumeSpecName: "kube-api-access-hhmmx") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "kube-api-access-hhmmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:21:06.365916 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.365899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:21:06.365986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.365933    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-token" (OuterVolumeSpecName: "istio-token") pod "60a5f141-951d-47ba-bf30-6d8a2be97fed" (UID: "60a5f141-951d-47ba-bf30-6d8a2be97fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:21:06.465063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.464963    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.465063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.465006    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.465063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.465021    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.465063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.465033    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.465063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.465047    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a5f141-951d-47ba-bf30-6d8a2be97fed-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.465063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.465059    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhmmx\" (UniqueName: \"kubernetes.io/projected/60a5f141-951d-47ba-bf30-6d8a2be97fed-kube-api-access-hhmmx\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:06.494314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.494285    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"]
Sep 15 09:21:06.496448 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:21:06.496416    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215e84d6_da28_47f4_83a4_1b9c61931183.slice/crio-8a10d68a9491bd14ea204919ffe146e17551842dd150fe0be801bdc2e16e934a WatchSource:0}: Error finding container 8a10d68a9491bd14ea204919ffe146e17551842dd150fe0be801bdc2e16e934a: Status 404 returned error can't find the container with id 8a10d68a9491bd14ea204919ffe146e17551842dd150fe0be801bdc2e16e934a
Sep 15 09:21:06.802716 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.802631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:06.889401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:06.889365    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:07.052711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.052678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:07.238132 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.238097    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8" event={"ID":"60a5f141-951d-47ba-bf30-6d8a2be97fed","Type":"ContainerDied","Data":"bfc8085d4fc63cd7cb401b3d870a8ca0ee647dc607dc5706ca561ff30989c169"}
Sep 15 09:21:07.238348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.238106    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"
Sep 15 09:21:07.238348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.238173    2571 scope.go:117] "RemoveContainer" containerID="bb90bbc54f5b134302794c77062ba627cfca605adaf440d9c6b3ae41603999e2"
Sep 15 09:21:07.240394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.240366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck" event={"ID":"215e84d6-da28-47f4-83a4-1b9c61931183","Type":"ContainerStarted","Data":"f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480"}
Sep 15 09:21:07.240526 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.240405    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck" event={"ID":"215e84d6-da28-47f4-83a4-1b9c61931183","Type":"ContainerStarted","Data":"8a10d68a9491bd14ea204919ffe146e17551842dd150fe0be801bdc2e16e934a"}
Sep 15 09:21:07.263389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.263327    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck" podStartSLOduration=2.263307617 podStartE2EDuration="2.263307617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:21:07.26156298 +0000 UTC m=+859.720852524" watchObservedRunningTime="2026-09-15 09:21:07.263307617 +0000 UTC m=+859.722597158"
Sep 15 09:21:07.278205 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.278169    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"]
Sep 15 09:21:07.280469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.280447    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:07.283886 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.283857    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-0cec7519-gw-openshift-default-6b58cc6b88-9fts8"]
Sep 15 09:21:07.370357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.370324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:07.772925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:07.772886    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:08.055608 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:08.055528    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" path="/var/lib/kubelet/pods/60a5f141-951d-47ba-bf30-6d8a2be97fed/volumes"
Sep 15 09:21:08.487037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:08.487003    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:09.191725 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:09.191688    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:09.930570 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:09.930534    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:15.312607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:15.312571    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:21:16.211986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:16.211944    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:16.212196 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:16.212006    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:16.226223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:16.226188    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:16.308841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:16.308806    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:21:26.802016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:26.801979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:27.841391 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:27.841358    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:29.470361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:29.470323    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:30.284133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.284099    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:30.691552 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.691517    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"]
Sep 15 09:21:30.691941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.691912    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy"
Sep 15 09:21:30.691941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.691924    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy"
Sep 15 09:21:30.691941 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.691939    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy"
Sep 15 09:21:30.692043 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.691944    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy"
Sep 15 09:21:30.692043 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.692021    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy"
Sep 15 09:21:30.692043 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.692030    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a5f141-951d-47ba-bf30-6d8a2be97fed" containerName="istio-proxy"
Sep 15 09:21:30.695517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.695498    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.698911 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.698887    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a23d9f\"/\"openshift-service-ca.crt\""
Sep 15 09:21:30.699044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.698891    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a23d9f\"/\"test-model-a23d9f-kserve-self-signed-certs\""
Sep 15 09:21:30.699044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.698939    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a23d9f\"/\"kube-root-ca.crt\""
Sep 15 09:21:30.699044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.698898    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a23d9f\"/\"default-dockercfg-986ng\""
Sep 15 09:21:30.703428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.703394    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"]
Sep 15 09:21:30.819171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.819132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-home\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.819171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.819176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250cec8-cd4b-428a-bed0-004cd78c2245-tls-certs\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.819387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.819200    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-tmp-dir\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.819387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.819322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-dshm\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.819387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.819376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-model-cache\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.819495 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.819402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhzf\" (UniqueName: \"kubernetes.io/projected/1250cec8-cd4b-428a-bed0-004cd78c2245-kube-api-access-4dhzf\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.920629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.920586    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-dshm\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.920808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.920646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-model-cache\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.920808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.920671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhzf\" (UniqueName: \"kubernetes.io/projected/1250cec8-cd4b-428a-bed0-004cd78c2245-kube-api-access-4dhzf\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.920808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.920736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-home\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.920808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.920758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250cec8-cd4b-428a-bed0-004cd78c2245-tls-certs\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.920808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.920779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-tmp-dir\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.921214 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.921186    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-model-cache\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.921292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.921221    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-tmp-dir\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.921292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.921280    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-home\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.923063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.923028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-dshm\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.923167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.923154    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250cec8-cd4b-428a-bed0-004cd78c2245-tls-certs\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:30.928357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:30.928330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhzf\" (UniqueName: \"kubernetes.io/projected/1250cec8-cd4b-428a-bed0-004cd78c2245-kube-api-access-4dhzf\") pod \"test-model-a23d9f-kserve-bd657dd67-lz6w8\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") " pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:31.006612 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.006514    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:31.265527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.265446    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:31.279103 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.279022    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"]
Sep 15 09:21:31.281418 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:21:31.281387    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1250cec8_cd4b_428a_bed0_004cd78c2245.slice/crio-e444ed3530a06b62798b7fcb7b0ce02ab89e4ee1403e298f8dda4836f1a9b7fd WatchSource:0}: Error finding container e444ed3530a06b62798b7fcb7b0ce02ab89e4ee1403e298f8dda4836f1a9b7fd: Status 404 returned error can't find the container with id e444ed3530a06b62798b7fcb7b0ce02ab89e4ee1403e298f8dda4836f1a9b7fd
Sep 15 09:21:31.373359 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.373321    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8" event={"ID":"1250cec8-cd4b-428a-bed0-004cd78c2245","Type":"ContainerStarted","Data":"dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7"}
Sep 15 09:21:31.373529 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.373368    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8" event={"ID":"1250cec8-cd4b-428a-bed0-004cd78c2245","Type":"ContainerStarted","Data":"e444ed3530a06b62798b7fcb7b0ce02ab89e4ee1403e298f8dda4836f1a9b7fd"}
Sep 15 09:21:31.391627 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.391564    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8" podStartSLOduration=1.391544368 podStartE2EDuration="1.391544368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:21:31.389441938 +0000 UTC m=+883.848731480" watchObservedRunningTime="2026-09-15 09:21:31.391544368 +0000 UTC m=+883.850833910"
Sep 15 09:21:31.416014 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:31.415979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:32.221179 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:32.221147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:33.065083 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:33.065054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:33.779605 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:33.779551    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:34.576507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:34.576456    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:35.429098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:35.429068    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:36.103555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:36.103516    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:36.727325 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:36.727293    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:36.816080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:36.816032    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:36.945139 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:36.945106    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:37.043968 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:37.043885    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:37.103331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:37.103297    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:37.460788 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:37.460757    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:38.175914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:38.175882    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:38.262814 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:38.262780    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:38.711688 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:38.711657    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:39.005008 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:39.004931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:39.096540 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:39.096499    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:39.230056 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:39.230027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:39.983890 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:39.983848    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:40.772181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:40.772151    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:41.007621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:41.007583    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:41.007621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:41.007632    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:41.020328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:41.020300    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:41.434473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:41.434442    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:21:41.440267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:41.440239    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:41.949812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:41.949779    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:42.050398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:42.050366    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:42.422531 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:42.422496    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:42.515925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:42.515819    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:21:43.101729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:43.101696    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:44.061735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:44.061680    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:44.766064 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:44.766031    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:44.819094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:44.819065    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:44.919614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:44.919556    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:45.223883 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:45.223849    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:45.324796 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:45.324767    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:45.693493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:45.693454    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:46.489560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:46.489529    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:47.178632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:47.178593    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:48.009197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:48.009163    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:21:48.009953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:48.009930    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:21:49.379177 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:49.379142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:49.955957 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:49.955920    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"]
Sep 15 09:21:49.956195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:49.956169    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" podUID="21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" containerName="maas-api" containerID="cri-o://62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829" gracePeriod=30
Sep 15 09:21:49.979722 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:49.979696    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:50.075169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.075133    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:50.175422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.175389    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:50.216284 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.216226    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:21:50.238510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.238481    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:50.332086 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.332043    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") pod \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") "
Sep 15 09:21:50.332548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.332510    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-metrics-certs\") pod \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") "
Sep 15 09:21:50.332667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.332616    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcmcz\" (UniqueName: \"kubernetes.io/projected/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-kube-api-access-bcmcz\") pod \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") "
Sep 15 09:21:50.332667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.332662    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-openshift-service-ca\") pod \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\" (UID: \"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841\") "
Sep 15 09:21:50.333341 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.333293    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" (UID: "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:21:50.335026 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.335003    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" (UID: "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:21:50.335139 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.335071    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" (UID: "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:21:50.335482 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.335457    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-kube-api-access-bcmcz" (OuterVolumeSpecName: "kube-api-access-bcmcz") pod "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" (UID: "21f24a65-0c3a-41f5-83b6-9b9c0b1c7841"). InnerVolumeSpecName "kube-api-access-bcmcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:21:50.434429 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.434385    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-metrics-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:50.434429 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.434421    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcmcz\" (UniqueName: \"kubernetes.io/projected/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-kube-api-access-bcmcz\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:50.434429 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.434436    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-openshift-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:50.434942 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.434448    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841-maas-api-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:21:50.471969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.471881    2571 generic.go:356] "Generic (PLEG): container finished" podID="21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" containerID="62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829" exitCode=0
Sep 15 09:21:50.471969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.471924    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"
Sep 15 09:21:50.472143 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.471971    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" event={"ID":"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841","Type":"ContainerDied","Data":"62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829"}
Sep 15 09:21:50.472143 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.472022    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9" event={"ID":"21f24a65-0c3a-41f5-83b6-9b9c0b1c7841","Type":"ContainerDied","Data":"e262c3a69c2f0f4d4a849520f54a1e0104d78ba58c204ef7cc5435ea9797a408"}
Sep 15 09:21:50.472143 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.472048    2571 scope.go:117] "RemoveContainer" containerID="62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829"
Sep 15 09:21:50.481415 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.481400    2571 scope.go:117] "RemoveContainer" containerID="62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829"
Sep 15 09:21:50.481686 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:21:50.481669    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829\": container with ID starting with 62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829 not found: ID does not exist" containerID="62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829"
Sep 15 09:21:50.481729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.481694    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829"} err="failed to get container status \"62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829\": rpc error: code = NotFound desc = could not find container \"62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829\": container with ID starting with 62a950c1e5883fa26fe1e02cbb0db7673244124720324e6abbfdc1694ceea829 not found: ID does not exist"
Sep 15 09:21:50.494082 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.494056    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"]
Sep 15 09:21:50.497680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.497654    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-abf07f-666bdd8db8-z2sv9"]
Sep 15 09:21:50.596599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.596554    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:50.699034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:50.698999    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:51.583482 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:51.583452    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:52.055933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:52.055817    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" path="/var/lib/kubelet/pods/21f24a65-0c3a-41f5-83b6-9b9c0b1c7841/volumes"
Sep 15 09:21:52.162567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:52.162536    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:52.284016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:52.283979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:52.969222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:52.969161    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:53.878081 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:53.878023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:54.558160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:54.558131    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:55.569952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:55.569910    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"]
Sep 15 09:21:55.665716 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:55.665685    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:55.918977 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:55.918941    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:56.737022 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:56.736992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:57.594193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:57.594149    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:58.473191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:58.473147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:21:59.230388 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:21:59.230346    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:00.007772 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:00.007736    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:00.983630 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:00.983587    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:01.733908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:01.733872    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:05.610516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:05.610485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:07.108892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:07.108857    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:22:07.283257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:07.283215    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:07.356911 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:07.356877    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:07.457109 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:07.457018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:07.522150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:07.522083    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:07.671382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:07.671343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:08.085033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:08.084998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:08.384401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:08.384273    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:08.483203 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:08.483157    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:08.585213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:08.585174    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:09.003872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:09.003798    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:09.794636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:09.794608    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:09.981571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:09.981540    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:10.589947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.589908    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck" podUID="215e84d6-da28-47f4-83a4-1b9c61931183" containerName="main" containerID="cri-o://f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480" gracePeriod=45
Sep 15 09:22:10.597704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.597667    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:10.843544 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.843483    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:22:10.935186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.935151    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:10.944094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944068    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215e84d6-da28-47f4-83a4-1b9c61931183-tls-certs\") pod \"215e84d6-da28-47f4-83a4-1b9c61931183\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") "
Sep 15 09:22:10.944261 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944150    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggj5q\" (UniqueName: \"kubernetes.io/projected/215e84d6-da28-47f4-83a4-1b9c61931183-kube-api-access-ggj5q\") pod \"215e84d6-da28-47f4-83a4-1b9c61931183\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") "
Sep 15 09:22:10.944261 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944196    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-tmp-dir\") pod \"215e84d6-da28-47f4-83a4-1b9c61931183\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") "
Sep 15 09:22:10.944261 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944234    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-home\") pod \"215e84d6-da28-47f4-83a4-1b9c61931183\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") "
Sep 15 09:22:10.944432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944306    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-model-cache\") pod \"215e84d6-da28-47f4-83a4-1b9c61931183\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") "
Sep 15 09:22:10.944432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944349    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-dshm\") pod \"215e84d6-da28-47f4-83a4-1b9c61931183\" (UID: \"215e84d6-da28-47f4-83a4-1b9c61931183\") "
Sep 15 09:22:10.944547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944475    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "215e84d6-da28-47f4-83a4-1b9c61931183" (UID: "215e84d6-da28-47f4-83a4-1b9c61931183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:10.944547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944488    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-home" (OuterVolumeSpecName: "home") pod "215e84d6-da28-47f4-83a4-1b9c61931183" (UID: "215e84d6-da28-47f4-83a4-1b9c61931183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:10.944667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944607    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-model-cache" (OuterVolumeSpecName: "model-cache") pod "215e84d6-da28-47f4-83a4-1b9c61931183" (UID: "215e84d6-da28-47f4-83a4-1b9c61931183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:10.944808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944784    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:10.944881 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944813    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:10.944881 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.944845    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:10.946507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.946485    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-dshm" (OuterVolumeSpecName: "dshm") pod "215e84d6-da28-47f4-83a4-1b9c61931183" (UID: "215e84d6-da28-47f4-83a4-1b9c61931183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:10.946600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.946505    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215e84d6-da28-47f4-83a4-1b9c61931183-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "215e84d6-da28-47f4-83a4-1b9c61931183" (UID: "215e84d6-da28-47f4-83a4-1b9c61931183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:22:10.946600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:10.946520    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215e84d6-da28-47f4-83a4-1b9c61931183-kube-api-access-ggj5q" (OuterVolumeSpecName: "kube-api-access-ggj5q") pod "215e84d6-da28-47f4-83a4-1b9c61931183" (UID: "215e84d6-da28-47f4-83a4-1b9c61931183"). InnerVolumeSpecName "kube-api-access-ggj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:22:11.046215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.046179    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggj5q\" (UniqueName: \"kubernetes.io/projected/215e84d6-da28-47f4-83a4-1b9c61931183-kube-api-access-ggj5q\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:11.046215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.046210    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215e84d6-da28-47f4-83a4-1b9c61931183-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:11.046215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.046220    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215e84d6-da28-47f4-83a4-1b9c61931183-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:11.369912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.369856    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:11.586637 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.586591    2571 generic.go:356] "Generic (PLEG): container finished" podID="215e84d6-da28-47f4-83a4-1b9c61931183" containerID="f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480" exitCode=0
Sep 15 09:22:11.587051 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.586730    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"
Sep 15 09:22:11.587680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.586753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck" event={"ID":"215e84d6-da28-47f4-83a4-1b9c61931183","Type":"ContainerDied","Data":"f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480"}
Sep 15 09:22:11.587891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.587872    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck" event={"ID":"215e84d6-da28-47f4-83a4-1b9c61931183","Type":"ContainerDied","Data":"8a10d68a9491bd14ea204919ffe146e17551842dd150fe0be801bdc2e16e934a"}
Sep 15 09:22:11.588019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.588005    2571 scope.go:117] "RemoveContainer" containerID="f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480"
Sep 15 09:22:11.602628 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.602607    2571 scope.go:117] "RemoveContainer" containerID="f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480"
Sep 15 09:22:11.602960 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:22:11.602939    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480\": container with ID starting with f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480 not found: ID does not exist" containerID="f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480"
Sep 15 09:22:11.603028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.602968    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480"} err="failed to get container status \"f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480\": rpc error: code = NotFound desc = could not find container \"f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480\": container with ID starting with f0f98dfd26f616d3016283b4d782d38a6ffeaf906d5fc89a9b2c09ee7463d480 not found: ID does not exist"
Sep 15 09:22:11.613705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.613679    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"]
Sep 15 09:22:11.615487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:11.615462    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-abf07f/autores-model-abf07f-kserve-6559cc9867-nd9ck"]
Sep 15 09:22:12.054966 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:12.054928    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215e84d6-da28-47f4-83a4-1b9c61931183" path="/var/lib/kubelet/pods/215e84d6-da28-47f4-83a4-1b9c61931183/volumes"
Sep 15 09:22:15.658176 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:15.658146    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:15.723481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:15.723426    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:15.804815 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:15.804784    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:15.902220 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:15.902190    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:16.002124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:16.002044    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:16.366650 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:16.366622    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:25.904600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:25.904559    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:22:28.505615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.505580    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"]
Sep 15 09:22:28.506211 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.506096    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215e84d6-da28-47f4-83a4-1b9c61931183" containerName="main"
Sep 15 09:22:28.506211 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.506117    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="215e84d6-da28-47f4-83a4-1b9c61931183" containerName="main"
Sep 15 09:22:28.506211 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.506131    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" containerName="maas-api"
Sep 15 09:22:28.506211 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.506137    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" containerName="maas-api"
Sep 15 09:22:28.506373 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.506219    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="215e84d6-da28-47f4-83a4-1b9c61931183" containerName="main"
Sep 15 09:22:28.506373 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.506229    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f24a65-0c3a-41f5-83b6-9b9c0b1c7841" containerName="maas-api"
Sep 15 09:22:28.509477 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.509450    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.511853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.511808    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffb593\"/\"premium-w0-ffb593-kserve-self-signed-certs\""
Sep 15 09:22:28.519190 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.519164    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"]
Sep 15 09:22:28.536245 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.536216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:28.617192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.617153    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-home\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.617368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.617234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-model-cache\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.617368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.617275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-dshm\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.617368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.617304    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-tmp-dir\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.617368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.617337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a434ac87-3ccc-4156-b0e2-2b53398137c9-tls-certs\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.617531 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.617425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmc9\" (UniqueName: \"kubernetes.io/projected/a434ac87-3ccc-4156-b0e2-2b53398137c9-kube-api-access-zdmc9\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.718532 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718500    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-model-cache\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.718532 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-dshm\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.718797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-tmp-dir\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.718797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a434ac87-3ccc-4156-b0e2-2b53398137c9-tls-certs\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.718797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmc9\" (UniqueName: \"kubernetes.io/projected/a434ac87-3ccc-4156-b0e2-2b53398137c9-kube-api-access-zdmc9\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.718797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-home\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.719120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.718934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-model-cache\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.719120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.719033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-tmp-dir\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.719120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.719076    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-home\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.720919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.720897    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-dshm\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.721174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.721157    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a434ac87-3ccc-4156-b0e2-2b53398137c9-tls-certs\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.727348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.727328    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmc9\" (UniqueName: \"kubernetes.io/projected/a434ac87-3ccc-4156-b0e2-2b53398137c9-kube-api-access-zdmc9\") pod \"premium-w0-ffb593-kserve-754d45d75f-qg8pv\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") " pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:28.819270 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:28.819177    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:29.091584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:29.091440    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"]
Sep 15 09:22:29.093893 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:22:29.093858    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda434ac87_3ccc_4156_b0e2_2b53398137c9.slice/crio-1909bd68081a260a797f518ebc192fa29d71e55457cefd6b73523fba7c62bb9f WatchSource:0}: Error finding container 1909bd68081a260a797f518ebc192fa29d71e55457cefd6b73523fba7c62bb9f: Status 404 returned error can't find the container with id 1909bd68081a260a797f518ebc192fa29d71e55457cefd6b73523fba7c62bb9f
Sep 15 09:22:29.687361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:29.687316    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv" event={"ID":"a434ac87-3ccc-4156-b0e2-2b53398137c9","Type":"ContainerStarted","Data":"2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d"}
Sep 15 09:22:29.687361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:29.687365    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv" event={"ID":"a434ac87-3ccc-4156-b0e2-2b53398137c9","Type":"ContainerStarted","Data":"1909bd68081a260a797f518ebc192fa29d71e55457cefd6b73523fba7c62bb9f"}
Sep 15 09:22:29.704698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:29.704644    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv" podStartSLOduration=1.704628237 podStartE2EDuration="1.704628237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:22:29.702779604 +0000 UTC m=+942.162069145" watchObservedRunningTime="2026-09-15 09:22:29.704628237 +0000 UTC m=+942.163917777"
Sep 15 09:22:32.508172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:32.508136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:32.746231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:32.746195    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"]
Sep 15 09:22:32.746502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:32.746456    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" containerID="cri-o://35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3" gracePeriod=30
Sep 15 09:22:32.768586 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:32.768499    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:33.217145 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:33.217104    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:33.548113 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:33.548011    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:33.856295 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:33.856260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:36.599512 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:36.599465    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:36.654731 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:36.654688    2571 patch_prober.go:28] interesting pod/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 09:22:36.654919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:36.654768    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 09:22:36.781030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:36.780993    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:36.879192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:36.879093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:37.456632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:37.456596    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:37.993102 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:37.993077    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:22:38.111972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.111933    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-certs\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.111987    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istiod-ca-cert\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112022    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkqcg\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-kube-api-access-zkqcg\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112077    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-envoy\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112120    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-socket\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112169    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-credential-socket\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112197    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-token\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-podinfo\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112260    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-data\") pod \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\" (UID: \"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e\") "
Sep 15 09:22:38.112424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112279    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:38.112626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112490    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:38.112626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112503    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:38.112626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112506    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:22:38.112787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112738    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-data" (OuterVolumeSpecName: "istio-data") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:38.112880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112806    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.112880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112857    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.112880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112872    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.113011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.112884    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.114421 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.114386    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:22:38.114548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.114534    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:22:38.114734 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.114717    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-token" (OuterVolumeSpecName: "istio-token") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:22:38.114816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.114800    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-kube-api-access-zkqcg" (OuterVolumeSpecName: "kube-api-access-zkqcg") pod "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" (UID: "39a0dccb-1cc9-4c07-8d3f-bdd953c3091e"). InnerVolumeSpecName "kube-api-access-zkqcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:22:38.214381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.214340    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqcg\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-kube-api-access-zkqcg\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.214381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.214373    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.214381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.214382    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.214381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.214390    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.214649 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.214398    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:22:38.265337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.265296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:38.733285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.733251    2571 generic.go:356] "Generic (PLEG): container finished" podID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerID="35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3" exitCode=0
Sep 15 09:22:38.733490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.733299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" event={"ID":"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e","Type":"ContainerDied","Data":"35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3"}
Sep 15 09:22:38.733490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.733310    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"
Sep 15 09:22:38.733490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.733340    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc" event={"ID":"39a0dccb-1cc9-4c07-8d3f-bdd953c3091e","Type":"ContainerDied","Data":"14dd7c8c3e7d32f386d85bcfcceb0588876c2457f7f2b253182b1633a14bc8a3"}
Sep 15 09:22:38.733490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.733362    2571 scope.go:117] "RemoveContainer" containerID="35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3"
Sep 15 09:22:38.743393 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.743374    2571 scope.go:117] "RemoveContainer" containerID="4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"
Sep 15 09:22:38.751500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.751481    2571 scope.go:117] "RemoveContainer" containerID="35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3"
Sep 15 09:22:38.751776 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:22:38.751754    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3\": container with ID starting with 35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3 not found: ID does not exist" containerID="35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3"
Sep 15 09:22:38.751820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.751785    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3"} err="failed to get container status \"35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3\": rpc error: code = NotFound desc = could not find container \"35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3\": container with ID starting with 35c50883d36601a0bfd956e5ad4f746cd2a9484eb893c6576d4322715e0617c3 not found: ID does not exist"
Sep 15 09:22:38.751820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.751803    2571 scope.go:117] "RemoveContainer" containerID="4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"
Sep 15 09:22:38.752094 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:22:38.752075    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184\": container with ID starting with 4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184 not found: ID does not exist" containerID="4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"
Sep 15 09:22:38.752144 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.752100    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184"} err="failed to get container status \"4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184\": rpc error: code = NotFound desc = could not find container \"4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184\": container with ID starting with 4a3577046078970703333bdf78e324b8a3ecaa045868e9224650f9312888d184 not found: ID does not exist"
Sep 15 09:22:38.760048 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.760022    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"]
Sep 15 09:22:38.763878 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.763854    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-abf07f-openshift-default-796b6f5978-8jvkc"]
Sep 15 09:22:38.819345 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.819307    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:38.819522 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.819364    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:38.832777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:38.832746    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:39.077138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.077052    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:39.749926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.749895    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"]
Sep 15 09:22:39.750278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.750265    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy"
Sep 15 09:22:39.750358 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.750279    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy"
Sep 15 09:22:39.750358 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.750298    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy"
Sep 15 09:22:39.750358 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.750303    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy"
Sep 15 09:22:39.750507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.750368    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy"
Sep 15 09:22:39.759643 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.759613    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:22:39.759847 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.759709    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:39.762279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.762248    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c28cf7-serving-cert\""
Sep 15 09:22:39.764731 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.764696    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"]
Sep 15 09:22:39.806715 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.806676    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:39.904854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.904796    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:39.931856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.931784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:39.932039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.931870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df902373-93d1-4ea6-a44e-1052921b6850-openshift-service-ca\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:39.932039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.931890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glsst\" (UniqueName: \"kubernetes.io/projected/df902373-93d1-4ea6-a44e-1052921b6850-kube-api-access-glsst\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:39.932039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:39.931941    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-metrics-certs\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.033231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.033131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-metrics-certs\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.033231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.033235    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.033425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.033273    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df902373-93d1-4ea6-a44e-1052921b6850-openshift-service-ca\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.033425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.033291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glsst\" (UniqueName: \"kubernetes.io/projected/df902373-93d1-4ea6-a44e-1052921b6850-kube-api-access-glsst\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.033425 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:22:40.033397    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-serving-cert: secret "maas-api-e2e-explicit-c28cf7-serving-cert" not found
Sep 15 09:22:40.033539 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:22:40.033486    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls podName:df902373-93d1-4ea6-a44e-1052921b6850 nodeName:}" failed. No retries permitted until 2026-09-15 09:22:40.533462957 +0000 UTC m=+952.992752481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls") pod "maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" (UID: "df902373-93d1-4ea6-a44e-1052921b6850") : secret "maas-api-e2e-explicit-c28cf7-serving-cert" not found
Sep 15 09:22:40.034008 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.033984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df902373-93d1-4ea6-a44e-1052921b6850-openshift-service-ca\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.035569 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.035544    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-metrics-certs\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.042603 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.042577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glsst\" (UniqueName: \"kubernetes.io/projected/df902373-93d1-4ea6-a44e-1052921b6850-kube-api-access-glsst\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.056394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.056363    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" path="/var/lib/kubelet/pods/39a0dccb-1cc9-4c07-8d3f-bdd953c3091e/volumes"
Sep 15 09:22:40.273461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.273429    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:40.374265 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.374234    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:40.538806 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.538764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.541302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.541270    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls\") pod \"maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.673640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.673543    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:40.966438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:40.966352    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"]
Sep 15 09:22:40.969537 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:22:40.969510    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf902373_93d1_4ea6_a44e_1052921b6850.slice/crio-1cdc1748e646d77de8d6675ccd75283429077695ebb1580ceef23e5ea3de9ed9 WatchSource:0}: Error finding container 1cdc1748e646d77de8d6675ccd75283429077695ebb1580ceef23e5ea3de9ed9: Status 404 returned error can't find the container with id 1cdc1748e646d77de8d6675ccd75283429077695ebb1580ceef23e5ea3de9ed9
Sep 15 09:22:41.752103 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:41.752066    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" event={"ID":"df902373-93d1-4ea6-a44e-1052921b6850","Type":"ContainerStarted","Data":"270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04"}
Sep 15 09:22:41.752505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:41.752121    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" event={"ID":"df902373-93d1-4ea6-a44e-1052921b6850","Type":"ContainerStarted","Data":"1cdc1748e646d77de8d6675ccd75283429077695ebb1580ceef23e5ea3de9ed9"}
Sep 15 09:22:41.752505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:41.752159    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:41.769013 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:41.768962    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" podStartSLOduration=2.1996548 podStartE2EDuration="2.768944847s" totalImagesPullingTime="569.290047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:22:41.767947279 +0000 UTC m=+954.227236832" watchObservedRunningTime="2026-09-15 09:22:41.768944847 +0000 UTC m=+954.228234382"
Sep 15 09:22:42.362737 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:42.362701    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:44.787907 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.787863    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"]
Sep 15 09:22:44.788555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.788533    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a0dccb-1cc9-4c07-8d3f-bdd953c3091e" containerName="istio-proxy"
Sep 15 09:22:44.791571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.791548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.793995 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.793975    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"premium-w4-890f3e-kserve-self-signed-certs\""
Sep 15 09:22:44.801271 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.801245    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"]
Sep 15 09:22:44.883119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.883080    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tls-certs\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.883321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.883131    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-dshm\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.883321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.883197    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tmp-dir\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.883321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.883267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-model-cache\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.883321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.883318    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6xz\" (UniqueName: \"kubernetes.io/projected/5a3d2dd1-03af-4c21-b9c0-d511cf199807-kube-api-access-qh6xz\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.883493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.883338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-home\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.983948 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.983906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-dshm\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.983955    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tmp-dir\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-model-cache\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984103    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6xz\" (UniqueName: \"kubernetes.io/projected/5a3d2dd1-03af-4c21-b9c0-d511cf199807-kube-api-access-qh6xz\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-home\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tls-certs\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984529 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984495    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tmp-dir\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984529    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-home\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.984688 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.984631    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-model-cache\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.986323 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.986302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-dshm\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.986638 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.986622    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tls-certs\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:44.992403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:44.992378    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6xz\" (UniqueName: \"kubernetes.io/projected/5a3d2dd1-03af-4c21-b9c0-d511cf199807-kube-api-access-qh6xz\") pod \"premium-w4-890f3e-kserve-56b4c566c9-4zlxd\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") " pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:45.093261 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:45.093202    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:45.102214 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:45.102187    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:45.377057 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:45.376849    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"]
Sep 15 09:22:45.379972 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:22:45.379943    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a3d2dd1_03af_4c21_b9c0_d511cf199807.slice/crio-46ff71816b45bbef565dda66e3250c95d1509cbf1746fd2db82cc9d89fa2bb8b WatchSource:0}: Error finding container 46ff71816b45bbef565dda66e3250c95d1509cbf1746fd2db82cc9d89fa2bb8b: Status 404 returned error can't find the container with id 46ff71816b45bbef565dda66e3250c95d1509cbf1746fd2db82cc9d89fa2bb8b
Sep 15 09:22:45.774302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:45.774206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" event={"ID":"5a3d2dd1-03af-4c21-b9c0-d511cf199807","Type":"ContainerStarted","Data":"b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64"}
Sep 15 09:22:45.774302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:45.774259    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" event={"ID":"5a3d2dd1-03af-4c21-b9c0-d511cf199807","Type":"ContainerStarted","Data":"46ff71816b45bbef565dda66e3250c95d1509cbf1746fd2db82cc9d89fa2bb8b"}
Sep 15 09:22:45.792120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:45.792067    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" podStartSLOduration=1.792050505 podStartE2EDuration="1.792050505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:22:45.790803886 +0000 UTC m=+958.250093426" watchObservedRunningTime="2026-09-15 09:22:45.792050505 +0000 UTC m=+958.251340045"
Sep 15 09:22:46.410743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:46.410704    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:22:47.762559 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:47.762523    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:22:48.056540 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:48.056455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:48.124572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:48.124543    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:48.189427 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:48.189392    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:48.288912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:48.288866    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:54.435219 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:54.435178    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:55.102530 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:55.102493    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:55.102743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:55.102546    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:55.115480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:55.115450    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:55.836108 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:55.836075    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:22:58.719134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:58.719101    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:22:58.820063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:22:58.820033    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:01.740091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:01.740048    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:02.207593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:02.207561    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:03.118599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:03.118559    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:04.171725 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:04.171677    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:04.271287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:04.271226    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:05.083306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:05.083271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:05.925928 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:05.925894    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:06.640343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:06.640304    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:10.049807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:10.049771    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:10.151690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:10.151615    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:10.246729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:10.246690    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:14.713371 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:14.713324    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:23:20.759849 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:20.759792    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:21.589820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:21.589788    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:27.363088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:27.363056    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:27.468065 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:27.468034    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:27.567379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:27.567340    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:27.663997 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:27.663913    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:28.380519 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:28.380472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:28.471474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:28.471436    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:28.571281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:28.571246    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:30.104850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.104798    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:30.706150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.706113    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"]
Sep 15 09:23:30.710002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.709981    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.713609 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.713580    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c28cf7\"/\"openshift-service-ca.crt\""
Sep 15 09:23:30.713767 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.713614    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c28cf7\"/\"default-dockercfg-4445t\""
Sep 15 09:23:30.713767 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.713621    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c28cf7\"/\"explicit-model-c28cf7-kserve-self-signed-certs\""
Sep 15 09:23:30.713767 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.713672    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c28cf7\"/\"kube-root-ca.crt\""
Sep 15 09:23:30.719355 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.719329    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"]
Sep 15 09:23:30.731654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.731627    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:30.772117 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.772069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4989\" (UniqueName: \"kubernetes.io/projected/e70daacd-4c2d-450c-9738-782428b22dcb-kube-api-access-c4989\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.772298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.772124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70daacd-4c2d-450c-9738-782428b22dcb-tls-certs\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.772298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.772199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-model-cache\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.772298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.772265    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-dshm\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.772402 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.772308    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-home\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.772402 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.772341    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-tmp-dir\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.873474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.873427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70daacd-4c2d-450c-9738-782428b22dcb-tls-certs\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.873474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.873477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-model-cache\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.873748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.873518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-dshm\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.873748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.873551    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-home\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.873748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.873594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-tmp-dir\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.873748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.873717    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4989\" (UniqueName: \"kubernetes.io/projected/e70daacd-4c2d-450c-9738-782428b22dcb-kube-api-access-c4989\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.874110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.874065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-model-cache\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.874303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.874122    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-tmp-dir\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.874427 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.874348    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-home\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.876688 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.876660    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70daacd-4c2d-450c-9738-782428b22dcb-tls-certs\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.876797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.876692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-dshm\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:30.880991 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:30.880971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4989\" (UniqueName: \"kubernetes.io/projected/e70daacd-4c2d-450c-9738-782428b22dcb-kube-api-access-c4989\") pod \"explicit-model-c28cf7-kserve-76766cb547-lgh9c\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") " pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:31.019910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:31.019816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:31.299561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:31.299377    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"]
Sep 15 09:23:31.302354 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:23:31.302323    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode70daacd_4c2d_450c_9738_782428b22dcb.slice/crio-ab561b688b7151bef121cf47274d48e7b5cc05213722f6474170be506115890b WatchSource:0}: Error finding container ab561b688b7151bef121cf47274d48e7b5cc05213722f6474170be506115890b: Status 404 returned error can't find the container with id ab561b688b7151bef121cf47274d48e7b5cc05213722f6474170be506115890b
Sep 15 09:23:32.004659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:32.004617    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c" event={"ID":"e70daacd-4c2d-450c-9738-782428b22dcb","Type":"ContainerStarted","Data":"8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29"}
Sep 15 09:23:32.004849 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:32.004671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c" event={"ID":"e70daacd-4c2d-450c-9738-782428b22dcb","Type":"ContainerStarted","Data":"ab561b688b7151bef121cf47274d48e7b5cc05213722f6474170be506115890b"}
Sep 15 09:23:32.023972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:32.023918    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c" podStartSLOduration=2.023902852 podStartE2EDuration="2.023902852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:23:32.022803857 +0000 UTC m=+1004.482093398" watchObservedRunningTime="2026-09-15 09:23:32.023902852 +0000 UTC m=+1004.483192393"
Sep 15 09:23:35.727014 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:35.726976    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:35.786883 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:35.786846    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:36.400507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:36.400469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:41.019959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:41.019917    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:41.020358 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:41.019973    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:41.032724 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:41.032698    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:41.072232 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:41.072202    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:23:42.428316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:42.428284    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:42.531581 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:42.531548    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:42.632900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:42.632865    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:42.941406 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:42.941373    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:43.277138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:43.277056    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:46.308589 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:46.308551    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:23:46.655690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:46.655652    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:46.930209 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:46.930116    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:47.027619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:47.027582    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:47.507385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:47.507347    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:48.573465 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:48.573432    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:49.277856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.277791    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:49.590217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.590181    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"]
Sep 15 09:23:49.594323 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.594304    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.596611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.596590    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"e2e-conflict-a-fd935fd3-kserve-self-signed-certs\""
Sep 15 09:23:49.602432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.602409    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"]
Sep 15 09:23:49.765959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.765918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-home\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.765959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.765962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2cf491-3d55-4181-88a8-98d42126ecd1-tls-certs\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.766184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.766032    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbzkd\" (UniqueName: \"kubernetes.io/projected/5e2cf491-3d55-4181-88a8-98d42126ecd1-kube-api-access-mbzkd\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.766184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.766055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-model-cache\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.766184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.766108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-tmp-dir\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.766184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.766158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-dshm\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867395 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbzkd\" (UniqueName: \"kubernetes.io/projected/5e2cf491-3d55-4181-88a8-98d42126ecd1-kube-api-access-mbzkd\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867395 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-model-cache\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867395 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867373    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-tmp-dir\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867676 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-dshm\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867676 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-home\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867676 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2cf491-3d55-4181-88a8-98d42126ecd1-tls-certs\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867815 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-model-cache\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867823    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-tmp-dir\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.867900 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.867885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-home\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.869798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.869775    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-dshm\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.870216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.870194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2cf491-3d55-4181-88a8-98d42126ecd1-tls-certs\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.875984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.875953    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbzkd\" (UniqueName: \"kubernetes.io/projected/5e2cf491-3d55-4181-88a8-98d42126ecd1-kube-api-access-mbzkd\") pod \"e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:49.905055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:49.905015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:50.171652 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:50.171527    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:50.178463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:50.178432    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"]
Sep 15 09:23:50.180204 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:23:50.180171    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e2cf491_3d55_4181_88a8_98d42126ecd1.slice/crio-c115d906ede9787e9b58ba96ffa26021801cbf514ac7201f06b89ac687d9d0c0 WatchSource:0}: Error finding container c115d906ede9787e9b58ba96ffa26021801cbf514ac7201f06b89ac687d9d0c0: Status 404 returned error can't find the container with id c115d906ede9787e9b58ba96ffa26021801cbf514ac7201f06b89ac687d9d0c0
Sep 15 09:23:50.272102 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:50.272055    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:50.893055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:50.893017    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:50.993433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:50.993388    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:51.112706 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:51.112671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92" event={"ID":"5e2cf491-3d55-4181-88a8-98d42126ecd1","Type":"ContainerStarted","Data":"d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748"}
Sep 15 09:23:51.112706 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:51.112712    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92" event={"ID":"5e2cf491-3d55-4181-88a8-98d42126ecd1","Type":"ContainerStarted","Data":"c115d906ede9787e9b58ba96ffa26021801cbf514ac7201f06b89ac687d9d0c0"}
Sep 15 09:23:51.132687 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:51.132633    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92" podStartSLOduration=2.1326165 podStartE2EDuration="2.1326165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:23:51.130218672 +0000 UTC m=+1023.589508214" watchObservedRunningTime="2026-09-15 09:23:51.1326165 +0000 UTC m=+1023.591906041"
Sep 15 09:23:51.463951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:51.463918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:51.567652 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:51.567616    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:51.722063 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:51.721981    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:55.285628 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:55.285591    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"]
Sep 15 09:23:55.313255 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:55.313227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:55.650001 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:55.649966    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:56.692912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:56.692877    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:57.477814 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:57.477783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:58.384972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:58.384935    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:59.075753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:59.075718    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:59.750103 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:59.750072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:23:59.905247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:59.905209    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:59.905423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:59.905260    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:23:59.918155 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:23:59.918129    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:24:00.170492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:00.170451    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:24:00.577632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:00.577503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:01.341002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:01.340972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:02.190329 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:02.190296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:03.079993 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:03.079959    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:03.793635 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:03.793603    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:04.616350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:04.616314    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:05.588464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:05.588427    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:06.436062 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:06.436018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:06.751999 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:06.751909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.173540 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.173505    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"]
Sep 15 09:24:07.173783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.173759    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" podUID="df902373-93d1-4ea6-a44e-1052921b6850" containerName="maas-api" containerID="cri-o://270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04" gracePeriod=30
Sep 15 09:24:07.197869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.197843    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.275374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.275339    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.378120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.378084    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.435884 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.435808    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:24:07.468291 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.468248    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.531789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.531753    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.550983 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.550955    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls\") pod \"df902373-93d1-4ea6-a44e-1052921b6850\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") "
Sep 15 09:24:07.551087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.551026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-metrics-certs\") pod \"df902373-93d1-4ea6-a44e-1052921b6850\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") "
Sep 15 09:24:07.551147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.551103    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df902373-93d1-4ea6-a44e-1052921b6850-openshift-service-ca\") pod \"df902373-93d1-4ea6-a44e-1052921b6850\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") "
Sep 15 09:24:07.551147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.551129    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glsst\" (UniqueName: \"kubernetes.io/projected/df902373-93d1-4ea6-a44e-1052921b6850-kube-api-access-glsst\") pod \"df902373-93d1-4ea6-a44e-1052921b6850\" (UID: \"df902373-93d1-4ea6-a44e-1052921b6850\") "
Sep 15 09:24:07.551768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.551739    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df902373-93d1-4ea6-a44e-1052921b6850-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df902373-93d1-4ea6-a44e-1052921b6850" (UID: "df902373-93d1-4ea6-a44e-1052921b6850"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:24:07.553239 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.553208    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df902373-93d1-4ea6-a44e-1052921b6850" (UID: "df902373-93d1-4ea6-a44e-1052921b6850"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:24:07.553239 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.553226    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df902373-93d1-4ea6-a44e-1052921b6850" (UID: "df902373-93d1-4ea6-a44e-1052921b6850"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:24:07.553365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.553293    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df902373-93d1-4ea6-a44e-1052921b6850-kube-api-access-glsst" (OuterVolumeSpecName: "kube-api-access-glsst") pod "df902373-93d1-4ea6-a44e-1052921b6850" (UID: "df902373-93d1-4ea6-a44e-1052921b6850"). InnerVolumeSpecName "kube-api-access-glsst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:24:07.652617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.652571    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-maas-api-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:07.652617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.652608    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df902373-93d1-4ea6-a44e-1052921b6850-metrics-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:07.652617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.652621    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df902373-93d1-4ea6-a44e-1052921b6850-openshift-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:07.652895 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.652633    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glsst\" (UniqueName: \"kubernetes.io/projected/df902373-93d1-4ea6-a44e-1052921b6850-kube-api-access-glsst\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:07.819744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.819666    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:07.919506 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:07.919469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:08.207006 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.206964    2571 generic.go:356] "Generic (PLEG): container finished" podID="df902373-93d1-4ea6-a44e-1052921b6850" containerID="270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04" exitCode=0
Sep 15 09:24:08.207195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.207032    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"
Sep 15 09:24:08.207195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.207045    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" event={"ID":"df902373-93d1-4ea6-a44e-1052921b6850","Type":"ContainerDied","Data":"270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04"}
Sep 15 09:24:08.207195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.207100    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv" event={"ID":"df902373-93d1-4ea6-a44e-1052921b6850","Type":"ContainerDied","Data":"1cdc1748e646d77de8d6675ccd75283429077695ebb1580ceef23e5ea3de9ed9"}
Sep 15 09:24:08.207195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.207120    2571 scope.go:117] "RemoveContainer" containerID="270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04"
Sep 15 09:24:08.216016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.215998    2571 scope.go:117] "RemoveContainer" containerID="270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04"
Sep 15 09:24:08.216297 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:24:08.216277    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04\": container with ID starting with 270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04 not found: ID does not exist" containerID="270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04"
Sep 15 09:24:08.216369 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.216309    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04"} err="failed to get container status \"270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04\": rpc error: code = NotFound desc = could not find container \"270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04\": container with ID starting with 270db71577c96e2f77d91096d1f8080626beb2f319faff289c5f0671c9ecaf04 not found: ID does not exist"
Sep 15 09:24:08.222312 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.222286    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"]
Sep 15 09:24:08.225855 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.225812    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c28cf7-75f8df754b-w6lzv"]
Sep 15 09:24:08.614055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:08.614021    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:09.506289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:09.506252    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:10.055727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.055690    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df902373-93d1-4ea6-a44e-1052921b6850" path="/var/lib/kubelet/pods/df902373-93d1-4ea6-a44e-1052921b6850/volumes"
Sep 15 09:24:10.306993 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.306889    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c" podUID="e70daacd-4c2d-450c-9738-782428b22dcb" containerName="main" containerID="cri-o://8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29" gracePeriod=45
Sep 15 09:24:10.307737 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.307546    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:10.568939 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.568878    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:24:10.685947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.685905    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-dshm\") pod \"e70daacd-4c2d-450c-9738-782428b22dcb\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") "
Sep 15 09:24:10.686150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.685968    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-home\") pod \"e70daacd-4c2d-450c-9738-782428b22dcb\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") "
Sep 15 09:24:10.686150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686004    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-model-cache\") pod \"e70daacd-4c2d-450c-9738-782428b22dcb\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") "
Sep 15 09:24:10.686150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686029    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-tmp-dir\") pod \"e70daacd-4c2d-450c-9738-782428b22dcb\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") "
Sep 15 09:24:10.686150 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686056    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4989\" (UniqueName: \"kubernetes.io/projected/e70daacd-4c2d-450c-9738-782428b22dcb-kube-api-access-c4989\") pod \"e70daacd-4c2d-450c-9738-782428b22dcb\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") "
Sep 15 09:24:10.686362 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686225    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70daacd-4c2d-450c-9738-782428b22dcb-tls-certs\") pod \"e70daacd-4c2d-450c-9738-782428b22dcb\" (UID: \"e70daacd-4c2d-450c-9738-782428b22dcb\") "
Sep 15 09:24:10.686362 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686258    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e70daacd-4c2d-450c-9738-782428b22dcb" (UID: "e70daacd-4c2d-450c-9738-782428b22dcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:24:10.686362 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686290    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-home" (OuterVolumeSpecName: "home") pod "e70daacd-4c2d-450c-9738-782428b22dcb" (UID: "e70daacd-4c2d-450c-9738-782428b22dcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:24:10.686362 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686302    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e70daacd-4c2d-450c-9738-782428b22dcb" (UID: "e70daacd-4c2d-450c-9738-782428b22dcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:24:10.686588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686572    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:10.686643 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686592    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:10.686643 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.686606    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:10.688287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.688266    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-dshm" (OuterVolumeSpecName: "dshm") pod "e70daacd-4c2d-450c-9738-782428b22dcb" (UID: "e70daacd-4c2d-450c-9738-782428b22dcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:24:10.688382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.688311    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70daacd-4c2d-450c-9738-782428b22dcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e70daacd-4c2d-450c-9738-782428b22dcb" (UID: "e70daacd-4c2d-450c-9738-782428b22dcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:24:10.688440 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.688393    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70daacd-4c2d-450c-9738-782428b22dcb-kube-api-access-c4989" (OuterVolumeSpecName: "kube-api-access-c4989") pod "e70daacd-4c2d-450c-9738-782428b22dcb" (UID: "e70daacd-4c2d-450c-9738-782428b22dcb"). InnerVolumeSpecName "kube-api-access-c4989". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:24:10.787370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.787328    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70daacd-4c2d-450c-9738-782428b22dcb-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:10.787370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.787361    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70daacd-4c2d-450c-9738-782428b22dcb-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:10.787370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:10.787371    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4989\" (UniqueName: \"kubernetes.io/projected/e70daacd-4c2d-450c-9738-782428b22dcb-kube-api-access-c4989\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:24:11.093085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.093054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:11.225840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.225803    2571 generic.go:356] "Generic (PLEG): container finished" podID="e70daacd-4c2d-450c-9738-782428b22dcb" containerID="8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29" exitCode=0
Sep 15 09:24:11.226028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.225866    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"
Sep 15 09:24:11.226028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.225902    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c" event={"ID":"e70daacd-4c2d-450c-9738-782428b22dcb","Type":"ContainerDied","Data":"8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29"}
Sep 15 09:24:11.226028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.225947    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c" event={"ID":"e70daacd-4c2d-450c-9738-782428b22dcb","Type":"ContainerDied","Data":"ab561b688b7151bef121cf47274d48e7b5cc05213722f6474170be506115890b"}
Sep 15 09:24:11.226028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.225964    2571 scope.go:117] "RemoveContainer" containerID="8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29"
Sep 15 09:24:11.235303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.235282    2571 scope.go:117] "RemoveContainer" containerID="8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29"
Sep 15 09:24:11.235607 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:24:11.235587    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29\": container with ID starting with 8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29 not found: ID does not exist" containerID="8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29"
Sep 15 09:24:11.235667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.235616    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29"} err="failed to get container status \"8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29\": rpc error: code = NotFound desc = could not find container \"8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29\": container with ID starting with 8f454723d864d8b9c5889515dfc18df5736e8430e32e4231c426d595bbde3e29 not found: ID does not exist"
Sep 15 09:24:11.247296 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.247265    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"]
Sep 15 09:24:11.255405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.250191    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c28cf7/explicit-model-c28cf7-kserve-76766cb547-lgh9c"]
Sep 15 09:24:11.783893 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:11.783851    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:12.055500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:12.055404    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70daacd-4c2d-450c-9738-782428b22dcb" path="/var/lib/kubelet/pods/e70daacd-4c2d-450c-9738-782428b22dcb/volumes"
Sep 15 09:24:12.699357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:12.699320    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:13.933194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:13.933158    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:14.031746 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:14.031695    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:14.153410 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:14.153378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:14.256265 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:14.256172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:14.593454 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:14.593417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:14.898663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:14.898577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:15.394036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:15.394005    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:15.841170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:15.841142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:16.279580 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:16.279493    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:16.692698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:16.692665    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:16.791704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:16.791665    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:16.915965 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:16.915933    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:17.111132 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:17.111099    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:18.077250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:18.077215    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:18.886978 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:18.886934    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:19.781447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:19.781417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:24.506300 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:24.506258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:24:28.747197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:28.747155    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:28.844408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:28.844375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:28.945041 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:28.945001    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:28.998577 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:28.998492    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:29.322641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:29.322556    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:29.423619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:29.423585    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:30.106436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:30.106402    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:30.976478 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:30.976446    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:31.674325 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:31.674292    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:32.381614 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:32.381577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:35.077939 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:35.077904    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:35.210121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:35.210066    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:35.365090 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:35.365055    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:35.702499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:35.702419    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:35.801610 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:35.801576    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:35.930875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:35.930845    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:41.661548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:41.661466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:41.723594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:41.723561    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:41.795607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:41.795575    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:41.894479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:41.894445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:42.041462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:42.041386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:47.627762 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:47.627722    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:47.727998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:47.727970    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:48.587427 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:48.587397    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:48.963288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:48.963199    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:49.164099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:49.164041    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:51.949724 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:51.949688    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:52.600206 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:52.600166    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:52.691563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:52.691532    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:52.792237 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:52.792205    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:53.683050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:53.683018    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:54.389580 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:54.389544    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:55.074126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:55.074077    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:55.204727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:55.204690    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:55.878446 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:55.878409    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:55.979357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:55.979324    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:56.081459 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:56.081422    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:56.347588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:56.347559    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:56.441729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:56.441697    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:24:58.606682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:24:58.606643    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:25:05.684111 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:05.684074    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"]
Sep 15 09:25:05.706263 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:05.706228    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:09.286751 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:09.286711    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:10.088984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:10.088945    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:10.778225 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:10.778191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:11.550353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:11.550314    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:12.284728 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:12.284690    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:13.169510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:13.169471    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:14.037558 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:14.037506    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:14.885071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:14.885029    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:15.462976 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.462944    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:15.563178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.563142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:15.690406 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.690374    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:15.888447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888412    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"]
Sep 15 09:25:15.888879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888854    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df902373-93d1-4ea6-a44e-1052921b6850" containerName="maas-api"
Sep 15 09:25:15.888879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888872    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="df902373-93d1-4ea6-a44e-1052921b6850" containerName="maas-api"
Sep 15 09:25:15.889036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888890    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70daacd-4c2d-450c-9738-782428b22dcb" containerName="main"
Sep 15 09:25:15.889036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888896    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70daacd-4c2d-450c-9738-782428b22dcb" containerName="main"
Sep 15 09:25:15.889036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888958    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="df902373-93d1-4ea6-a44e-1052921b6850" containerName="maas-api"
Sep 15 09:25:15.889036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.888967    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70daacd-4c2d-450c-9738-782428b22dcb" containerName="main"
Sep 15 09:25:15.892060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.892042    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:15.894572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.894550    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-981036\"/\"default-dockercfg-g9dpx\""
Sep 15 09:25:15.894683 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.894599    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-981036\"/\"kube-root-ca.crt\""
Sep 15 09:25:15.895675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.895656    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-981036\"/\"noten-model-981036-kserve-self-signed-certs\""
Sep 15 09:25:15.895734 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.895689    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-981036\"/\"openshift-service-ca.crt\""
Sep 15 09:25:15.899596 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.899565    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"]
Sep 15 09:25:15.982059 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.982020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4p46\" (UniqueName: \"kubernetes.io/projected/1de9946a-d6af-40bd-b394-611aa44dd0c5-kube-api-access-j4p46\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:15.982223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.982099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de9946a-d6af-40bd-b394-611aa44dd0c5-tls-certs\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:15.982223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.982149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-tmp-dir\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:15.982223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.982214    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-dshm\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:15.982352 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.982245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-home\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:15.982352 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:15.982259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-model-cache\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.082791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.082753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-dshm\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.082972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.082801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-home\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.082972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.082817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-model-cache\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.082972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.082903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4p46\" (UniqueName: \"kubernetes.io/projected/1de9946a-d6af-40bd-b394-611aa44dd0c5-kube-api-access-j4p46\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.083134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.082979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de9946a-d6af-40bd-b394-611aa44dd0c5-tls-certs\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.083134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.083013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-tmp-dir\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.083324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.083295    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-home\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.083410 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.083332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-tmp-dir\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.083480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.083453    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-model-cache\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.085128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.085101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-dshm\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.085471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.085452    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de9946a-d6af-40bd-b394-611aa44dd0c5-tls-certs\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.091998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.091975    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4p46\" (UniqueName: \"kubernetes.io/projected/1de9946a-d6af-40bd-b394-611aa44dd0c5-kube-api-access-j4p46\") pod \"noten-model-981036-kserve-74c57cbf87-xp68p\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") " pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.202991 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.202889    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:16.307250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.307219    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:16.476069 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.475999    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"]
Sep 15 09:25:16.477718 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:25:16.477671    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1de9946a_d6af_40bd_b394_611aa44dd0c5.slice/crio-30b7e18dced9a00f486d8efff0f409d5e739f4366fe2f7a1d1e860aed3b25390 WatchSource:0}: Error finding container 30b7e18dced9a00f486d8efff0f409d5e739f4366fe2f7a1d1e860aed3b25390: Status 404 returned error can't find the container with id 30b7e18dced9a00f486d8efff0f409d5e739f4366fe2f7a1d1e860aed3b25390
Sep 15 09:25:16.565899 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.565861    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p" event={"ID":"1de9946a-d6af-40bd-b394-611aa44dd0c5","Type":"ContainerStarted","Data":"372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201"}
Sep 15 09:25:16.565899 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.565904    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p" event={"ID":"1de9946a-d6af-40bd-b394-611aa44dd0c5","Type":"ContainerStarted","Data":"30b7e18dced9a00f486d8efff0f409d5e739f4366fe2f7a1d1e860aed3b25390"}
Sep 15 09:25:16.582197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.582142    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p" podStartSLOduration=1.582125117 podStartE2EDuration="1.582125117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:25:16.581156398 +0000 UTC m=+1109.040445952" watchObservedRunningTime="2026-09-15 09:25:16.582125117 +0000 UTC m=+1109.041414657"
Sep 15 09:25:16.695885 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:16.695854    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:17.582854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:17.582792    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:17.685308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:17.685268    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:17.956291 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:17.956201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:18.252430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.252348    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"]
Sep 15 09:25:18.256032 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.256010    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.258466 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.258440    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f032a1-openshift-default-dockercfg-fs4zj\""
Sep 15 09:25:18.267550 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.267520    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"]
Sep 15 09:25:18.276374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.276347    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:18.408584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408551    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prn6p\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-kube-api-access-prn6p\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.408741 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56abe022-36c3-4004-91a8-ed166248ea47-istiod-ca-cert\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.408741 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-envoy\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.408915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-data\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.408915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408782    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56abe022-36c3-4004-91a8-ed166248ea47-istio-podinfo\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.408915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-certs\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.408915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408903    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-socket\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.409108 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.408962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-credential-socket\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.409108 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.409048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-istio-token\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.409641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.409612    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:18.509879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.509754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56abe022-36c3-4004-91a8-ed166248ea47-istio-podinfo\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.509879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.509804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-certs\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.509879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.509851    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-socket\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.509879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.509881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-credential-socket\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.509934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-istio-token\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.509970    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prn6p\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-kube-api-access-prn6p\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56abe022-36c3-4004-91a8-ed166248ea47-istiod-ca-cert\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510047    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-envoy\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-data\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510255    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-certs\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-credential-socket\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510368    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-socket\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510441    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-data\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.510961 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.510935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56abe022-36c3-4004-91a8-ed166248ea47-istiod-ca-cert\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.512373 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.512345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-envoy\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.512640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.512611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56abe022-36c3-4004-91a8-ed166248ea47-istio-podinfo\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.519599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.519573    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prn6p\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-kube-api-access-prn6p\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.519700 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.519581    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-istio-token\") pod \"e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") " pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.568120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.568066    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:18.699238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.699206    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:18.844916 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.844885    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"]
Sep 15 09:25:18.846726 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:25:18.846693    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56abe022_36c3_4004_91a8_ed166248ea47.slice/crio-a75fbbda8189e71c5199860a5c93651f70322746c653f13aa5e7ec399373a752 WatchSource:0}: Error finding container a75fbbda8189e71c5199860a5c93651f70322746c653f13aa5e7ec399373a752: Status 404 returned error can't find the container with id a75fbbda8189e71c5199860a5c93651f70322746c653f13aa5e7ec399373a752
Sep 15 09:25:18.849456 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.849409    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:25:18.849553 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.849508    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:25:18.849594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:18.849555    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:25:19.317052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:19.316931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:19.582976 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:19.582945    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" event={"ID":"56abe022-36c3-4004-91a8-ed166248ea47","Type":"ContainerStarted","Data":"25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"}
Sep 15 09:25:19.583162 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:19.582994    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" event={"ID":"56abe022-36c3-4004-91a8-ed166248ea47","Type":"ContainerStarted","Data":"a75fbbda8189e71c5199860a5c93651f70322746c653f13aa5e7ec399373a752"}
Sep 15 09:25:19.602617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:19.602564    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podStartSLOduration=1.602548096 podStartE2EDuration="1.602548096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:25:19.600295996 +0000 UTC m=+1112.059585587" watchObservedRunningTime="2026-09-15 09:25:19.602548096 +0000 UTC m=+1112.061837636"
Sep 15 09:25:19.985234 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:19.985141    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"]
Sep 15 09:25:19.986066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:19.986030    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" podUID="ad89f547-857b-46ba-ae32-7e9f960fb392" containerName="maas-api" containerID="cri-o://982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31" gracePeriod=30
Sep 15 09:25:20.008845 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.008802    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.103621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.103590    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.163002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.162931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.245127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.245063    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:25:20.267233 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.267174    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.362740 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.362707    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.433048 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.433003    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls\") pod \"ad89f547-857b-46ba-ae32-7e9f960fb392\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") "
Sep 15 09:25:20.433048 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.433047    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-metrics-certs\") pod \"ad89f547-857b-46ba-ae32-7e9f960fb392\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") "
Sep 15 09:25:20.433331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.433172    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwz8s\" (UniqueName: \"kubernetes.io/projected/ad89f547-857b-46ba-ae32-7e9f960fb392-kube-api-access-bwz8s\") pod \"ad89f547-857b-46ba-ae32-7e9f960fb392\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") "
Sep 15 09:25:20.433331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.433206    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89f547-857b-46ba-ae32-7e9f960fb392-openshift-service-ca\") pod \"ad89f547-857b-46ba-ae32-7e9f960fb392\" (UID: \"ad89f547-857b-46ba-ae32-7e9f960fb392\") "
Sep 15 09:25:20.433632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.433602    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad89f547-857b-46ba-ae32-7e9f960fb392-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad89f547-857b-46ba-ae32-7e9f960fb392" (UID: "ad89f547-857b-46ba-ae32-7e9f960fb392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:25:20.435308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.435279    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad89f547-857b-46ba-ae32-7e9f960fb392-kube-api-access-bwz8s" (OuterVolumeSpecName: "kube-api-access-bwz8s") pod "ad89f547-857b-46ba-ae32-7e9f960fb392" (UID: "ad89f547-857b-46ba-ae32-7e9f960fb392"). InnerVolumeSpecName "kube-api-access-bwz8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:25:20.435714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.435694    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad89f547-857b-46ba-ae32-7e9f960fb392" (UID: "ad89f547-857b-46ba-ae32-7e9f960fb392"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:25:20.435781 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.435718    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad89f547-857b-46ba-ae32-7e9f960fb392" (UID: "ad89f547-857b-46ba-ae32-7e9f960fb392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:25:20.534793 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.534696    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwz8s\" (UniqueName: \"kubernetes.io/projected/ad89f547-857b-46ba-ae32-7e9f960fb392-kube-api-access-bwz8s\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:20.534793 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.534728    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89f547-857b-46ba-ae32-7e9f960fb392-openshift-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:20.534793 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.534743    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-maas-api-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:20.534793 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.534755    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad89f547-857b-46ba-ae32-7e9f960fb392-metrics-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:20.568676 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.568635    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:20.574143 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.574113    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:20.589369 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.589338    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad89f547-857b-46ba-ae32-7e9f960fb392" containerID="982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31" exitCode=0
Sep 15 09:25:20.589542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.589378    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"
Sep 15 09:25:20.589542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.589422    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" event={"ID":"ad89f547-857b-46ba-ae32-7e9f960fb392","Type":"ContainerDied","Data":"982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31"}
Sep 15 09:25:20.589542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.589470    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w" event={"ID":"ad89f547-857b-46ba-ae32-7e9f960fb392","Type":"ContainerDied","Data":"03f018558752c0f90236049292d5c4482ece9e870da5532fafdaf3c415537659"}
Sep 15 09:25:20.589542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.589501    2571 scope.go:117] "RemoveContainer" containerID="982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31"
Sep 15 09:25:20.589820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.589798    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:20.590822 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.590730    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:20.600707 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.600689    2571 scope.go:117] "RemoveContainer" containerID="982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31"
Sep 15 09:25:20.601298 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:25:20.601220    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31\": container with ID starting with 982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31 not found: ID does not exist" containerID="982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31"
Sep 15 09:25:20.601298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.601257    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31"} err="failed to get container status \"982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31\": rpc error: code = NotFound desc = could not find container \"982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31\": container with ID starting with 982d2d6461f0da878426b944dac66fdfa635d315c132d418ef52520848038f31 not found: ID does not exist"
Sep 15 09:25:20.631403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.631365    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"]
Sep 15 09:25:20.637133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.637103    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a23d9f-76c795bb67-5vf7w"]
Sep 15 09:25:20.662976 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.662926    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.706511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.706472    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8" podUID="1250cec8-cd4b-428a-bed0-004cd78c2245" containerName="main" containerID="cri-o://dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7" gracePeriod=45
Sep 15 09:25:20.891455 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.891425    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:20.970085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:20.970042    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:25:21.142532 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142481    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-dshm\") pod \"1250cec8-cd4b-428a-bed0-004cd78c2245\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") "
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142553    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-model-cache\") pod \"1250cec8-cd4b-428a-bed0-004cd78c2245\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") "
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142581    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-home\") pod \"1250cec8-cd4b-428a-bed0-004cd78c2245\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") "
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142617    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250cec8-cd4b-428a-bed0-004cd78c2245-tls-certs\") pod \"1250cec8-cd4b-428a-bed0-004cd78c2245\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") "
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142816    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dhzf\" (UniqueName: \"kubernetes.io/projected/1250cec8-cd4b-428a-bed0-004cd78c2245-kube-api-access-4dhzf\") pod \"1250cec8-cd4b-428a-bed0-004cd78c2245\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") "
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142934    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-model-cache" (OuterVolumeSpecName: "model-cache") pod "1250cec8-cd4b-428a-bed0-004cd78c2245" (UID: "1250cec8-cd4b-428a-bed0-004cd78c2245"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.142951    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-home" (OuterVolumeSpecName: "home") pod "1250cec8-cd4b-428a-bed0-004cd78c2245" (UID: "1250cec8-cd4b-428a-bed0-004cd78c2245"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:25:21.143035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.143032    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-tmp-dir\") pod \"1250cec8-cd4b-428a-bed0-004cd78c2245\" (UID: \"1250cec8-cd4b-428a-bed0-004cd78c2245\") "
Sep 15 09:25:21.143379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.143288    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1250cec8-cd4b-428a-bed0-004cd78c2245" (UID: "1250cec8-cd4b-428a-bed0-004cd78c2245"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:25:21.143438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.143422    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:21.143494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.143439    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:21.143494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.143449    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:21.145033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.145008    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-dshm" (OuterVolumeSpecName: "dshm") pod "1250cec8-cd4b-428a-bed0-004cd78c2245" (UID: "1250cec8-cd4b-428a-bed0-004cd78c2245"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:25:21.145285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.145254    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1250cec8-cd4b-428a-bed0-004cd78c2245-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1250cec8-cd4b-428a-bed0-004cd78c2245" (UID: "1250cec8-cd4b-428a-bed0-004cd78c2245"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:25:21.145285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.145261    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1250cec8-cd4b-428a-bed0-004cd78c2245-kube-api-access-4dhzf" (OuterVolumeSpecName: "kube-api-access-4dhzf") pod "1250cec8-cd4b-428a-bed0-004cd78c2245" (UID: "1250cec8-cd4b-428a-bed0-004cd78c2245"). InnerVolumeSpecName "kube-api-access-4dhzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:25:21.244629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.244584    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250cec8-cd4b-428a-bed0-004cd78c2245-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:21.244629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.244617    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250cec8-cd4b-428a-bed0-004cd78c2245-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:21.244629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.244630    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhzf\" (UniqueName: \"kubernetes.io/projected/1250cec8-cd4b-428a-bed0-004cd78c2245-kube-api-access-4dhzf\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:25:21.596460 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.596426    2571 generic.go:356] "Generic (PLEG): container finished" podID="1250cec8-cd4b-428a-bed0-004cd78c2245" containerID="dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7" exitCode=0
Sep 15 09:25:21.596658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.596467    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"
Sep 15 09:25:21.596658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.596512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8" event={"ID":"1250cec8-cd4b-428a-bed0-004cd78c2245","Type":"ContainerDied","Data":"dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7"}
Sep 15 09:25:21.596658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.596566    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8" event={"ID":"1250cec8-cd4b-428a-bed0-004cd78c2245","Type":"ContainerDied","Data":"e444ed3530a06b62798b7fcb7b0ce02ab89e4ee1403e298f8dda4836f1a9b7fd"}
Sep 15 09:25:21.596658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.596588    2571 scope.go:117] "RemoveContainer" containerID="dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7"
Sep 15 09:25:21.607173 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.607154    2571 scope.go:117] "RemoveContainer" containerID="dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7"
Sep 15 09:25:21.607483 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:25:21.607464    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7\": container with ID starting with dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7 not found: ID does not exist" containerID="dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7"
Sep 15 09:25:21.607539 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.607492    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7"} err="failed to get container status \"dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7\": rpc error: code = NotFound desc = could not find container \"dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7\": container with ID starting with dca44ba2537e82fdc74083c1f7cb60df9994ca8cffe687bc2f49cea291f547f7 not found: ID does not exist"
Sep 15 09:25:21.619851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.619809    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"]
Sep 15 09:25:21.623477 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.623450    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a23d9f/test-model-a23d9f-kserve-bd657dd67-lz6w8"]
Sep 15 09:25:21.765197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.765164    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:21.915000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:21.914914    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:22.056407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:22.056371    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1250cec8-cd4b-428a-bed0-004cd78c2245" path="/var/lib/kubelet/pods/1250cec8-cd4b-428a-bed0-004cd78c2245/volumes"
Sep 15 09:25:22.056777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:22.056763    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad89f547-857b-46ba-ae32-7e9f960fb392" path="/var/lib/kubelet/pods/ad89f547-857b-46ba-ae32-7e9f960fb392/volumes"
Sep 15 09:25:22.722383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:22.722347    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:23.781345 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:23.781309    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:24.389375 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:24.389332    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241021    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"]
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241646    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad89f547-857b-46ba-ae32-7e9f960fb392" containerName="maas-api"
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241665    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad89f547-857b-46ba-ae32-7e9f960fb392" containerName="maas-api"
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241695    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1250cec8-cd4b-428a-bed0-004cd78c2245" containerName="main"
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241704    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1250cec8-cd4b-428a-bed0-004cd78c2245" containerName="main"
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241798    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad89f547-857b-46ba-ae32-7e9f960fb392" containerName="maas-api"
Sep 15 09:25:25.242129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.241814    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1250cec8-cd4b-428a-bed0-004cd78c2245" containerName="main"
Sep 15 09:25:25.246972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.246949    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.250689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.250660    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f032a1-serving-cert\""
Sep 15 09:25:25.250689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.250681    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 09:25:25.250934 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.250799    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 09:25:25.251018 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.250672    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ptrgq\""
Sep 15 09:25:25.251073 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.251045    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 09:25:25.261489 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.255796    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"]
Sep 15 09:25:25.281065 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.281030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:25.286299 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.286269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmb8\" (UniqueName: \"kubernetes.io/projected/f6a33b64-f722-4977-b84e-3392a87ed266-kube-api-access-tdmb8\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.286463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.286335    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a33b64-f722-4977-b84e-3392a87ed266-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.286463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.286421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.286581 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.286480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.381470 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.381437    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:25.387792 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.387767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.387946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.387813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.387946 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.387918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmb8\" (UniqueName: \"kubernetes.io/projected/f6a33b64-f722-4977-b84e-3392a87ed266-kube-api-access-tdmb8\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.388055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.387969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a33b64-f722-4977-b84e-3392a87ed266-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.388055 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:25:25.387977    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-serving-cert: secret "maas-api-e2e-shared-b-w2-f032a1-serving-cert" not found
Sep 15 09:25:25.388055 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:25:25.388038    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls podName:f6a33b64-f722-4977-b84e-3392a87ed266 nodeName:}" failed. No retries permitted until 2026-09-15 09:25:25.888021802 +0000 UTC m=+1118.347311321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" (UID: "f6a33b64-f722-4977-b84e-3392a87ed266") : secret "maas-api-e2e-shared-b-w2-f032a1-serving-cert" not found
Sep 15 09:25:25.388690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.388663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a33b64-f722-4977-b84e-3392a87ed266-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.390563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.390540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.396208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.396181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmb8\" (UniqueName: \"kubernetes.io/projected/f6a33b64-f722-4977-b84e-3392a87ed266-kube-api-access-tdmb8\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.481184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.481147    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:25.796645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.796607    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:25.893251 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.893216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:25.895486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.895446    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:25.895790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:25.895770    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:26.170268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.170229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:26.203657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.203604    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:26.203814 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.203678    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:26.216760 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.216726    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:26.289467 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.289431    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:26.445069 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.445042    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"]
Sep 15 09:25:26.447261 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:25:26.447230    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a33b64_f722_4977_b84e_3392a87ed266.slice/crio-642e6675b6ca4d09d08548a1d1d53d2e8c4682dc430a2c048265a9ed19904563 WatchSource:0}: Error finding container 642e6675b6ca4d09d08548a1d1d53d2e8c4682dc430a2c048265a9ed19904563: Status 404 returned error can't find the container with id 642e6675b6ca4d09d08548a1d1d53d2e8c4682dc430a2c048265a9ed19904563
Sep 15 09:25:26.448520 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.448497    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:25:26.624990 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.624951    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" event={"ID":"f6a33b64-f722-4977-b84e-3392a87ed266","Type":"ContainerStarted","Data":"642e6675b6ca4d09d08548a1d1d53d2e8c4682dc430a2c048265a9ed19904563"}
Sep 15 09:25:26.636272 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:26.636248    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:25:27.176361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.176277    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:27.393005 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.392972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:27.493626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.493548    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:27.634404 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.634368    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" event={"ID":"f6a33b64-f722-4977-b84e-3392a87ed266","Type":"ContainerStarted","Data":"89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5"}
Sep 15 09:25:27.634671 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.634645    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:27.652493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.652439    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" podStartSLOduration=2.195993675 podStartE2EDuration="2.652422559s" totalImagesPullingTime="456.428884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:25:27.651328872 +0000 UTC m=+1120.110618414" watchObservedRunningTime="2026-09-15 09:25:27.652422559 +0000 UTC m=+1120.111712094"
Sep 15 09:25:27.661372 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:27.661339    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:28.132987 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:28.132952    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:29.094745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:29.094706    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:29.605981 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:29.605949    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:25:30.087272 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:30.087232    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:30.893678 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:30.893635    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:31.084408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:31.084375    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:31.189931 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:31.189810    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:31.761431 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:31.761396    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:32.588684 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:32.588636    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:33.287112 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:33.287075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:33.645148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:33.645113    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:25:38.551797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:38.551766    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:38.622462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:38.622425    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:38.722947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:38.722909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:38.793347 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:38.793309    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:39.115185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:39.115150    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:39.215710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:39.215678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:39.965682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:39.965636    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:40.887616 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:40.887578    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:41.678322 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:41.678243    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:42.387364 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:42.387330    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:43.149663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:43.149631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:43.877875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:43.877819    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:44.136700 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:44.136595    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:44.709354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:44.709323    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:45.391972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:45.391926    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:45.993639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:45.993598    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"]
Sep 15 09:25:45.997147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:45.997130    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:45.999673 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:45.999656    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"e2e-conflict-b-fd935fd3-kserve-self-signed-certs\""
Sep 15 09:25:46.006660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.006634    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"]
Sep 15 09:25:46.036853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.035701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-model-cache\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.036853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.035803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pbh\" (UniqueName: \"kubernetes.io/projected/02e3c0ac-8430-41ab-a257-962ef793a3a5-kube-api-access-w7pbh\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.036853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.035870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-tmp-dir\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.036853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.035918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-home\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.036853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.035946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e3c0ac-8430-41ab-a257-962ef793a3a5-tls-certs\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.036853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.036004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-dshm\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.136728 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.136673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pbh\" (UniqueName: \"kubernetes.io/projected/02e3c0ac-8430-41ab-a257-962ef793a3a5-kube-api-access-w7pbh\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137008    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-tmp-dir\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137254 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-home\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e3c0ac-8430-41ab-a257-962ef793a3a5-tls-certs\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137534    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-dshm\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137641    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-home\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137441    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-tmp-dir\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.137897 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.137874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-model-cache\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.145853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.144286    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-model-cache\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.145853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.144415    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-dshm\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.146722 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.146691    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e3c0ac-8430-41ab-a257-962ef793a3a5-tls-certs\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.146969 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.146945    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pbh\" (UniqueName: \"kubernetes.io/projected/02e3c0ac-8430-41ab-a257-962ef793a3a5-kube-api-access-w7pbh\") pod \"e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") " pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.293289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.293199    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:46.308384 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.308349    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:46.394872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.394819    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:46.606459 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.606424    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"]
Sep 15 09:25:46.609620 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:25:46.609587    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02e3c0ac_8430_41ab_a257_962ef793a3a5.slice/crio-f2c4d5a325641f0b2f3a935a42cbd7530349c2c92eea633fd68706a9caa203b7 WatchSource:0}: Error finding container f2c4d5a325641f0b2f3a935a42cbd7530349c2c92eea633fd68706a9caa203b7: Status 404 returned error can't find the container with id f2c4d5a325641f0b2f3a935a42cbd7530349c2c92eea633fd68706a9caa203b7
Sep 15 09:25:46.619513 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.619417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:46.753356 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.753313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx" event={"ID":"02e3c0ac-8430-41ab-a257-962ef793a3a5","Type":"ContainerStarted","Data":"19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46"}
Sep 15 09:25:46.753545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.753363    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx" event={"ID":"02e3c0ac-8430-41ab-a257-962ef793a3a5","Type":"ContainerStarted","Data":"f2c4d5a325641f0b2f3a935a42cbd7530349c2c92eea633fd68706a9caa203b7"}
Sep 15 09:25:46.771172 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.771119    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx" podStartSLOduration=1.77109918 podStartE2EDuration="1.77109918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:25:46.76951967 +0000 UTC m=+1139.228809213" watchObservedRunningTime="2026-09-15 09:25:46.77109918 +0000 UTC m=+1139.230388721"
Sep 15 09:25:47.000038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:46.999951    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:49.966318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:49.966201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:50.204274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.204244    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:50.316310 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.316235    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:50.592126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.592087    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 09:25:50.592332 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.592147    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 09:25:50.777583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.777491    2571 generic.go:356] "Generic (PLEG): container finished" podID="56abe022-36c3-4004-91a8-ed166248ea47" containerID="25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812" exitCode=0
Sep 15 09:25:50.777583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.777567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" event={"ID":"56abe022-36c3-4004-91a8-ed166248ea47","Type":"ContainerDied","Data":"25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"}
Sep 15 09:25:50.778054 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.778037    2571 scope.go:117] "RemoveContainer" containerID="25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"
Sep 15 09:25:50.779179 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.779152    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:25:50.779240 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.779220    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:25:50.779276 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.779250    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:25:50.814393 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:50.814361    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:25:51.403399 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:51.403368    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:51.784978 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:51.784889    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" event={"ID":"56abe022-36c3-4004-91a8-ed166248ea47","Type":"ContainerStarted","Data":"abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146"}
Sep 15 09:25:52.568277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:52.568237    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:52.568735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:52.568475    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 09:25:52.568735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:52.568546    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 09:25:53.569498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:53.569453    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 09:25:53.569923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:53.569512    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 09:25:54.569268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:54.569229    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 09:25:54.569444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:54.569286    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 09:25:55.299573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:55.299533    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"]
Sep 15 09:25:55.345744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:55.345692    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:55.569008 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:55.568914    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 09:25:55.569008 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:55.568969    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 09:25:56.309053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:56.309012    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:56.309053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:56.309062    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:56.347243 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:56.347210    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:56.568618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:56.568522    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 15 09:25:56.568618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:56.568591    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 15 09:25:56.821935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:56.821819    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:25:57.572764 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:57.572730    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:57.573182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:57.573143    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:57.573899 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:57.573882    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:25:58.037194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:58.037164    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:58.328197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:58.328117    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:25:58.724097 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:25:58.724058    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:00.267217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:00.267182    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:01.071695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:01.071662    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:01.172279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:01.172246    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:01.972904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:01.972872    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:03.001585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:03.001551    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:03.691417 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:03.691386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:04.452639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:04.452604    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:04.553330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:04.553294    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:04.653028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:04.652995    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:04.931059 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:04.931016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:05.031224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:05.031195    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:08.693601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:08.693570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:10.319271 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.319180    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p" podUID="1de9946a-d6af-40bd-b394-611aa44dd0c5" containerName="main" containerID="cri-o://372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201" gracePeriod=45
Sep 15 09:26:10.560741 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.560710    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:26:10.718312 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718264    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4p46\" (UniqueName: \"kubernetes.io/projected/1de9946a-d6af-40bd-b394-611aa44dd0c5-kube-api-access-j4p46\") pod \"1de9946a-d6af-40bd-b394-611aa44dd0c5\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") "
Sep 15 09:26:10.718312 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718331    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-model-cache\") pod \"1de9946a-d6af-40bd-b394-611aa44dd0c5\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") "
Sep 15 09:26:10.718593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718360    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de9946a-d6af-40bd-b394-611aa44dd0c5-tls-certs\") pod \"1de9946a-d6af-40bd-b394-611aa44dd0c5\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") "
Sep 15 09:26:10.718593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718478    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-home\") pod \"1de9946a-d6af-40bd-b394-611aa44dd0c5\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") "
Sep 15 09:26:10.718593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718525    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-dshm\") pod \"1de9946a-d6af-40bd-b394-611aa44dd0c5\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") "
Sep 15 09:26:10.718593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718572    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-tmp-dir\") pod \"1de9946a-d6af-40bd-b394-611aa44dd0c5\" (UID: \"1de9946a-d6af-40bd-b394-611aa44dd0c5\") "
Sep 15 09:26:10.718807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718630    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1de9946a-d6af-40bd-b394-611aa44dd0c5" (UID: "1de9946a-d6af-40bd-b394-611aa44dd0c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:10.718915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718851    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-home" (OuterVolumeSpecName: "home") pod "1de9946a-d6af-40bd-b394-611aa44dd0c5" (UID: "1de9946a-d6af-40bd-b394-611aa44dd0c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:10.718915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.718903    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1de9946a-d6af-40bd-b394-611aa44dd0c5" (UID: "1de9946a-d6af-40bd-b394-611aa44dd0c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:10.719046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.719033    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:10.719095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.719049    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:10.719095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.719062    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:10.720738 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.720707    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de9946a-d6af-40bd-b394-611aa44dd0c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1de9946a-d6af-40bd-b394-611aa44dd0c5" (UID: "1de9946a-d6af-40bd-b394-611aa44dd0c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:26:10.720982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.720949    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-dshm" (OuterVolumeSpecName: "dshm") pod "1de9946a-d6af-40bd-b394-611aa44dd0c5" (UID: "1de9946a-d6af-40bd-b394-611aa44dd0c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:10.721064 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.720998    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de9946a-d6af-40bd-b394-611aa44dd0c5-kube-api-access-j4p46" (OuterVolumeSpecName: "kube-api-access-j4p46") pod "1de9946a-d6af-40bd-b394-611aa44dd0c5" (UID: "1de9946a-d6af-40bd-b394-611aa44dd0c5"). InnerVolumeSpecName "kube-api-access-j4p46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:26:10.820479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.820431    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4p46\" (UniqueName: \"kubernetes.io/projected/1de9946a-d6af-40bd-b394-611aa44dd0c5-kube-api-access-j4p46\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:10.820479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.820475    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de9946a-d6af-40bd-b394-611aa44dd0c5-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:10.820479 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.820486    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de9946a-d6af-40bd-b394-611aa44dd0c5-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:10.887504 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.887470    2571 generic.go:356] "Generic (PLEG): container finished" podID="1de9946a-d6af-40bd-b394-611aa44dd0c5" containerID="372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201" exitCode=0
Sep 15 09:26:10.887695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.887535    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"
Sep 15 09:26:10.887695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.887555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p" event={"ID":"1de9946a-d6af-40bd-b394-611aa44dd0c5","Type":"ContainerDied","Data":"372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201"}
Sep 15 09:26:10.887695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.887601    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p" event={"ID":"1de9946a-d6af-40bd-b394-611aa44dd0c5","Type":"ContainerDied","Data":"30b7e18dced9a00f486d8efff0f409d5e739f4366fe2f7a1d1e860aed3b25390"}
Sep 15 09:26:10.887695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.887620    2571 scope.go:117] "RemoveContainer" containerID="372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201"
Sep 15 09:26:10.896955 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.896930    2571 scope.go:117] "RemoveContainer" containerID="372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201"
Sep 15 09:26:10.897258 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:26:10.897239    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201\": container with ID starting with 372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201 not found: ID does not exist" containerID="372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201"
Sep 15 09:26:10.897328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.897266    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201"} err="failed to get container status \"372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201\": rpc error: code = NotFound desc = could not find container \"372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201\": container with ID starting with 372359ab9c2ef5d91fc077ef9daf95237ac5352f806b15fff4a3f05b35d77201 not found: ID does not exist"
Sep 15 09:26:10.908377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.908348    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"]
Sep 15 09:26:10.910240 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:10.910221    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-981036/noten-model-981036-kserve-74c57cbf87-xp68p"]
Sep 15 09:26:12.055256 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:12.055219    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de9946a-d6af-40bd-b394-611aa44dd0c5" path="/var/lib/kubelet/pods/1de9946a-d6af-40bd-b394-611aa44dd0c5/volumes"
Sep 15 09:26:13.204418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:13.204390    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:13.413502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:13.413463    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:26:21.166399 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:21.166363    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:21.388233 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:21.388197    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"]
Sep 15 09:26:21.388460 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:21.388436    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" containerID="cri-o://72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586" gracePeriod=30
Sep 15 09:26:21.413511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:21.413477    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:21.996681 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:21.996644    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:22.294499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.292261    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"]
Sep 15 09:26:22.294499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.293256    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de9946a-d6af-40bd-b394-611aa44dd0c5" containerName="main"
Sep 15 09:26:22.294499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.293278    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de9946a-d6af-40bd-b394-611aa44dd0c5" containerName="main"
Sep 15 09:26:22.294499 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.293542    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de9946a-d6af-40bd-b394-611aa44dd0c5" containerName="main"
Sep 15 09:26:22.297711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.297680    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.300859 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.300819    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f79800-openshift-default-dockercfg-p5fzd\""
Sep 15 09:26:22.307648 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.307042    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"]
Sep 15 09:26:22.319260 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.319171    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:22.351937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.351885    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-credential-socket\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.351997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-certs\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795tj\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-kube-api-access-795tj\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-envoy\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-data\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-socket\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352202    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istiod-ca-cert\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352238    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-podinfo\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.352264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.352256    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-token\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453253 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453210    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-certs\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795tj\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-kube-api-access-795tj\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453322    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-envoy\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-data\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-socket\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453678 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istiod-ca-cert\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453678 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453614    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-podinfo\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453678 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-token\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453675    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-certs\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-credential-socket\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453732    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-data\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.453865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453750    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-socket\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.454089 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.453949    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-credential-socket\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.454306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.454282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istiod-ca-cert\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.455766 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.455744    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-envoy\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.456132 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.456110    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-podinfo\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.461582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.461556    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-token\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.461790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.461768    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-795tj\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-kube-api-access-795tj\") pod \"e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") " pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.607807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.607767    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:22.619267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.619237    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:22.902198 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.902172    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"]
Sep 15 09:26:22.904534 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:26:22.904499    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0881a8_e2ba_42f7_bdea_20ec581d6bef.slice/crio-54eabb7e2318f33e08f8103b2f0b4098b8ca31f400f52a495081ccec02825874 WatchSource:0}: Error finding container 54eabb7e2318f33e08f8103b2f0b4098b8ca31f400f52a495081ccec02825874: Status 404 returned error can't find the container with id 54eabb7e2318f33e08f8103b2f0b4098b8ca31f400f52a495081ccec02825874
Sep 15 09:26:22.907009 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.906972    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:22.907089 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.907044    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:22.907089 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.907082    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:22.952441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:22.952411    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" event={"ID":"2f0881a8-e2ba-42f7-bdea-20ec581d6bef","Type":"ContainerStarted","Data":"54eabb7e2318f33e08f8103b2f0b4098b8ca31f400f52a495081ccec02825874"}
Sep 15 09:26:23.963750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:23.963710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" event={"ID":"2f0881a8-e2ba-42f7-bdea-20ec581d6bef","Type":"ContainerStarted","Data":"3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"}
Sep 15 09:26:23.984495 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:23.984442    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podStartSLOduration=1.984422302 podStartE2EDuration="1.984422302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:26:23.98165296 +0000 UTC m=+1176.440942504" watchObservedRunningTime="2026-09-15 09:26:23.984422302 +0000 UTC m=+1176.443711842"
Sep 15 09:26:24.619913 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:24.619871    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:24.625333 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:24.625304    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:24.900874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:24.900774    2571 patch_prober.go:28] interesting pod/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 09:26:24.900874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:24.900841    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 09:26:24.970060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:24.970033    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:24.971179 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:24.971163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:25.274440 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:25.274326    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:25.914102 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:25.914037    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:26.005808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.005772    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:26.142219 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.142188    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:26.644116 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.644085    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:26:26.699110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hl57\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-kube-api-access-7hl57\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699065    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-envoy\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699116    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-certs\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-data\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699188    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-socket\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699206    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-istio-token\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699276    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-credential-socket\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699311    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045d9d4-268e-4f08-a808-54faf133fae4-istio-podinfo\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699340    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045d9d4-268e-4f08-a808-54faf133fae4-istiod-ca-cert\") pod \"2045d9d4-268e-4f08-a808-54faf133fae4\" (UID: \"2045d9d4-268e-4f08-a808-54faf133fae4\") "
Sep 15 09:26:26.699634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699460    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:26.699634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:26.699634 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699489    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-data" (OuterVolumeSpecName: "istio-data") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:26.699866 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699713    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:26.699866 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699790    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2045d9d4-268e-4f08-a808-54faf133fae4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:26:26.699866 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699811    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.699866 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699862    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.700081 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699877    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.700081 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.699890    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.701372 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.701346    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-istio-token" (OuterVolumeSpecName: "istio-token") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:26:26.701488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.701383    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-kube-api-access-7hl57" (OuterVolumeSpecName: "kube-api-access-7hl57") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "kube-api-access-7hl57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:26:26.701488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.701392    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:26:26.701585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.701505    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2045d9d4-268e-4f08-a808-54faf133fae4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2045d9d4-268e-4f08-a808-54faf133fae4" (UID: "2045d9d4-268e-4f08-a808-54faf133fae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:26:26.801367 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.801328    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.801367 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.801363    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2045d9d4-268e-4f08-a808-54faf133fae4-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.801367 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.801374    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2045d9d4-268e-4f08-a808-54faf133fae4-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.801593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.801383    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hl57\" (UniqueName: \"kubernetes.io/projected/2045d9d4-268e-4f08-a808-54faf133fae4-kube-api-access-7hl57\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.801593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.801392    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2045d9d4-268e-4f08-a808-54faf133fae4-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:26:26.981268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.981181    2571 generic.go:356] "Generic (PLEG): container finished" podID="2045d9d4-268e-4f08-a808-54faf133fae4" containerID="72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586" exitCode=0
Sep 15 09:26:26.981268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.981227    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"
Sep 15 09:26:26.981268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.981257    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" event={"ID":"2045d9d4-268e-4f08-a808-54faf133fae4","Type":"ContainerDied","Data":"72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586"}
Sep 15 09:26:26.981493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.981313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c" event={"ID":"2045d9d4-268e-4f08-a808-54faf133fae4","Type":"ContainerDied","Data":"5c9171ad9a41a7512482ff6f9ca7af59af7008e173c071d9e09f0d6695a56d6a"}
Sep 15 09:26:26.981493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.981341    2571 scope.go:117] "RemoveContainer" containerID="72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586"
Sep 15 09:26:26.996164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:26.996133    2571 scope.go:117] "RemoveContainer" containerID="8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"
Sep 15 09:26:27.005669 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.005648    2571 scope.go:117] "RemoveContainer" containerID="72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586"
Sep 15 09:26:27.006010 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:26:27.005989    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586\": container with ID starting with 72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586 not found: ID does not exist" containerID="72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586"
Sep 15 09:26:27.006339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.006018    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586"} err="failed to get container status \"72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586\": rpc error: code = NotFound desc = could not find container \"72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586\": container with ID starting with 72c768eb63f00aaee20de0c54573b6c37e814a0ef4f25ff191804824d2508586 not found: ID does not exist"
Sep 15 09:26:27.006339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.006036    2571 scope.go:117] "RemoveContainer" containerID="8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"
Sep 15 09:26:27.006426 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:26:27.006355    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf\": container with ID starting with 8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf not found: ID does not exist" containerID="8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"
Sep 15 09:26:27.006426 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.006386    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf"} err="failed to get container status \"8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf\": rpc error: code = NotFound desc = could not find container \"8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf\": container with ID starting with 8578c575bf73d344e4af33d558cb9651a26ea482a904d40b8954d46eed3e2acf not found: ID does not exist"
Sep 15 09:26:27.008316 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.008283    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"]
Sep 15 09:26:27.011123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.011102    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2430e2-openshift-default-54fd5f9b88-96f4c"]
Sep 15 09:26:27.077754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.077718    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:27.887389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:27.887359    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:28.055309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:28.055273    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" path="/var/lib/kubelet/pods/2045d9d4-268e-4f08-a808-54faf133fae4/volumes"
Sep 15 09:26:28.748349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:28.748313    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:29.747565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:29.747534    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:30.585436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:30.585402    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:30.686402 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:30.686369    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:30.785711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:30.785678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:31.074370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:31.074284    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:31.176963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:31.176931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:31.259931 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:31.259903    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:31.819361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:31.819326    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:32.609415 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:32.609376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:32.860924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:32.860815    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:33.412869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:33.412812    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:34.096262 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:34.096227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:34.774734 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:34.774693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:35.567514 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:35.567484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:38.470480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:38.470441    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:38.534630 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:38.534597    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:47.311392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:47.311345    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:26:48.045002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:48.044965    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:26:48.047293 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:48.047263    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:26:52.663474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:52.663434    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:53.501564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:53.501527    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:54.177270 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:54.177234    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:54.862689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:54.862652    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:55.566765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:55.566724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:56.012050 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.011991    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 09:26:56.012310 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.012083    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 09:26:56.110757 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.110714    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:56.390067 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.390027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:56.490507 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.490469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:56.777222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777125    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"]
Sep 15 09:26:56.777675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777528    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy"
Sep 15 09:26:56.777675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777540    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy"
Sep 15 09:26:56.777675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777560    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy"
Sep 15 09:26:56.777675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777565    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy"
Sep 15 09:26:56.777675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777668    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy"
Sep 15 09:26:56.777994 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.777684    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2045d9d4-268e-4f08-a808-54faf133fae4" containerName="istio-proxy"
Sep 15 09:26:56.782110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.782086    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.784933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.784911    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8dd3bd-openshift-default-dockercfg-45mph\""
Sep 15 09:26:56.791070 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.791047    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"]
Sep 15 09:26:56.804028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.803964    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:56.929987 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.929952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-socket\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.929987 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.929989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-credential-socket\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-certs\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-data\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930141    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rb4\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-kube-api-access-g9rb4\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930191    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-envoy\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930257 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-istio-token\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f41605-709c-4a82-96f3-8b285971b171-istio-podinfo\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:56.930527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:56.930305    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f41605-709c-4a82-96f3-8b285971b171-istiod-ca-cert\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032021 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.031917    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f41605-709c-4a82-96f3-8b285971b171-istio-podinfo\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032021 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.031971    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f41605-709c-4a82-96f3-8b285971b171-istiod-ca-cert\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032030    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-socket\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-credential-socket\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032079    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-certs\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-data\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rb4\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-kube-api-access-g9rb4\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-envoy\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032247    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-istio-token\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-credential-socket\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032517    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-certs\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.032788 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.032669    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-data\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.033052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.033030    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-socket\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.033243 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.033225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f41605-709c-4a82-96f3-8b285971b171-istiod-ca-cert\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.034501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.034480    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f41605-709c-4a82-96f3-8b285971b171-istio-podinfo\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.034602 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.034535    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-envoy\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.051441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.051400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-istio-token\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.051739 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.051710    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rb4\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-kube-api-access-g9rb4\") pod \"e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") " pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.094807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.094759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:57.227019 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.226980    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:57.329353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.329275    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:57.380751 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.380603    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"]
Sep 15 09:26:57.383129 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:26:57.383094    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f41605_709c_4a82_96f3_8b285971b171.slice/crio-b1dd2831000acfdcc48da7526b367784af2a634bd1477b91c90b8180aa51f4fa WatchSource:0}: Error finding container b1dd2831000acfdcc48da7526b367784af2a634bd1477b91c90b8180aa51f4fa: Status 404 returned error can't find the container with id b1dd2831000acfdcc48da7526b367784af2a634bd1477b91c90b8180aa51f4fa
Sep 15 09:26:57.385685 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.385653    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:57.385759 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.385720    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:57.385759 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:57.385747    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:58.160645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.160609    2571 generic.go:356] "Generic (PLEG): container finished" podID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerID="3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b" exitCode=0
Sep 15 09:26:58.161137 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.160684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" event={"ID":"2f0881a8-e2ba-42f7-bdea-20ec581d6bef","Type":"ContainerDied","Data":"3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"}
Sep 15 09:26:58.161200 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.161160    2571 scope.go:117] "RemoveContainer" containerID="3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"
Sep 15 09:26:58.162378 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.162351    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" event={"ID":"32f41605-709c-4a82-96f3-8b285971b171","Type":"ContainerStarted","Data":"3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"}
Sep 15 09:26:58.162472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.162400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" event={"ID":"32f41605-709c-4a82-96f3-8b285971b171","Type":"ContainerStarted","Data":"b1dd2831000acfdcc48da7526b367784af2a634bd1477b91c90b8180aa51f4fa"}
Sep 15 09:26:58.162472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.162411    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:58.162551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.162479    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:58.162551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.162514    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:26:58.205704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:58.205643    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podStartSLOduration=2.205620351 podStartE2EDuration="2.205620351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:26:58.202457634 +0000 UTC m=+1210.661747176" watchObservedRunningTime="2026-09-15 09:26:58.205620351 +0000 UTC m=+1210.664909893"
Sep 15 09:26:59.096000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.095959    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:59.101027 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.100996    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:59.169491 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.169456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" event={"ID":"2f0881a8-e2ba-42f7-bdea-20ec581d6bef","Type":"ContainerStarted","Data":"161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6"}
Sep 15 09:26:59.169892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.169844    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:59.170535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.170518    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:26:59.303167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.303134    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"]
Sep 15 09:26:59.305970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.305947    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.308770 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.308751    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ffb593\"/\"distinct-2-w0-ffb593-kserve-self-signed-certs\""
Sep 15 09:26:59.315504 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.315477    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"]
Sep 15 09:26:59.402403 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.402292    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:59.461582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.461540    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-home\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.461768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.461611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtmw\" (UniqueName: \"kubernetes.io/projected/ea195e13-0b07-427d-9183-8c44672ed1fc-kube-api-access-hdtmw\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.461768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.461666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-model-cache\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.461768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.461733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea195e13-0b07-427d-9183-8c44672ed1fc-tls-certs\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.461970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.461799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-tmp-dir\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.461970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.461906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-dshm\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.562988 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.562941    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-tmp-dir\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-dshm\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-home\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtmw\" (UniqueName: \"kubernetes.io/projected/ea195e13-0b07-427d-9183-8c44672ed1fc-kube-api-access-hdtmw\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-model-cache\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea195e13-0b07-427d-9183-8c44672ed1fc-tls-certs\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563431 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563384    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-tmp-dir\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-home\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.563582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.563562    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-model-cache\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.565565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.565540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-dshm\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.565726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.565707    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea195e13-0b07-427d-9183-8c44672ed1fc-tls-certs\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.571264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.571241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtmw\" (UniqueName: \"kubernetes.io/projected/ea195e13-0b07-427d-9183-8c44672ed1fc-kube-api-access-hdtmw\") pod \"distinct-2-w0-ffb593-kserve-78c879b969-cxmbv\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") " pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.616013 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.615971    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:26:59.620593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.620563    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:26:59.620744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.620686    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 09:26:59.620744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.620731    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 09:26:59.709430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.709354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:26:59.907944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:26:59.907912    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"]
Sep 15 09:26:59.909479 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:26:59.909444    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea195e13_0b07_427d_9183_8c44672ed1fc.slice/crio-c7a70da7b3e50fc8c77f279d39f12afda4c1861fe42d77ad817eb84a8c1bcf8c WatchSource:0}: Error finding container c7a70da7b3e50fc8c77f279d39f12afda4c1861fe42d77ad817eb84a8c1bcf8c: Status 404 returned error can't find the container with id c7a70da7b3e50fc8c77f279d39f12afda4c1861fe42d77ad817eb84a8c1bcf8c
Sep 15 09:27:00.106819 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.106783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:00.177270 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.177236    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv" event={"ID":"ea195e13-0b07-427d-9183-8c44672ed1fc","Type":"ContainerStarted","Data":"3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec"}
Sep 15 09:27:00.177270 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.177277    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv" event={"ID":"ea195e13-0b07-427d-9183-8c44672ed1fc","Type":"ContainerStarted","Data":"c7a70da7b3e50fc8c77f279d39f12afda4c1861fe42d77ad817eb84a8c1bcf8c"}
Sep 15 09:27:00.193972 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.193915    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv" podStartSLOduration=1.193895355 podStartE2EDuration="1.193895355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:27:00.193675152 +0000 UTC m=+1212.652964694" watchObservedRunningTime="2026-09-15 09:27:00.193895355 +0000 UTC m=+1212.653184887"
Sep 15 09:27:00.201898 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.201813    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:00.276067 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.276040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:00.620576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.620517    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 09:27:00.620786 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:00.620604    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 09:27:01.189119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:01.189081    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:01.619721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:01.619668    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 09:27:01.619983 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:01.619738    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 09:27:01.913429 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:01.913353    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:02.620288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:02.620242    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 09:27:02.620789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:02.620313    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 09:27:02.750617 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:02.750583    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:03.450045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:03.450001    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:03.620510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:03.620473    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 15 09:27:03.621136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:03.621096    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 15 09:27:04.163133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:04.163096    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:04.623462 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:04.623428    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:27:04.623959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:04.623717    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:27:04.624425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:04.624408    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:27:04.995997 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:04.995920    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:05.054867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:05.054812    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:05.331550 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:05.331472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:05.502468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:05.502433    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:05.600710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:05.600677    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:09.011310 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:09.011257    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:27:09.616863 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:09.616811    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:27:09.617037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:09.616882    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:27:09.629574 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:09.629547    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:27:10.240788 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:10.240754    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:27:19.288889 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.288846    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"]
Sep 15 09:27:19.292301 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.292281    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.295842 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.295813    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fcb750c6\"/\"openshift-service-ca.crt\""
Sep 15 09:27:19.295984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.295855    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fcb750c6\"/\"e2e-lifecycle-model-fcb750c6-kserve-self-signed-certs\""
Sep 15 09:27:19.295984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.295904    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fcb750c6\"/\"default-dockercfg-t8rpp\""
Sep 15 09:27:19.295984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.295906    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fcb750c6\"/\"kube-root-ca.crt\""
Sep 15 09:27:19.302097 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.302073    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"]
Sep 15 09:27:19.377191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.377158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-tmp-dir\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.377191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.377193    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-home\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.377390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.377213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-dshm\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.377390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.377257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-model-cache\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.377390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.377281    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86jpm\" (UniqueName: \"kubernetes.io/projected/97632201-fdd8-4686-857e-04c4dcb8eddc-kube-api-access-86jpm\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.377497 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.377396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97632201-fdd8-4686-857e-04c4dcb8eddc-tls-certs\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.401319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.401288    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:19.478467 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.478428    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97632201-fdd8-4686-857e-04c4dcb8eddc-tls-certs\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.478660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.478564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-tmp-dir\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.478660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.478594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-home\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.478660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.478615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-dshm\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.478660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.478644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-model-cache\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.478890 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.478680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jpm\" (UniqueName: \"kubernetes.io/projected/97632201-fdd8-4686-857e-04c4dcb8eddc-kube-api-access-86jpm\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.479173 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.479136    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-home\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.479302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.479167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-model-cache\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.479302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.479194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-tmp-dir\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.481622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.481597    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-dshm\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.482237 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.482216    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97632201-fdd8-4686-857e-04c4dcb8eddc-tls-certs\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.486166 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.486146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jpm\" (UniqueName: \"kubernetes.io/projected/97632201-fdd8-4686-857e-04c4dcb8eddc-kube-api-access-86jpm\") pod \"e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") " pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.602983 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.602939    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:19.880731 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:19.880640    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"]
Sep 15 09:27:19.883654 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:27:19.883618    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97632201_fdd8_4686_857e_04c4dcb8eddc.slice/crio-87d185f12801723ae0d7eed9603d278425a120425e36a87d4ecca4d8797629bc WatchSource:0}: Error finding container 87d185f12801723ae0d7eed9603d278425a120425e36a87d4ecca4d8797629bc: Status 404 returned error can't find the container with id 87d185f12801723ae0d7eed9603d278425a120425e36a87d4ecca4d8797629bc
Sep 15 09:27:20.288238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:20.288139    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn" event={"ID":"97632201-fdd8-4686-857e-04c4dcb8eddc","Type":"ContainerStarted","Data":"b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1"}
Sep 15 09:27:20.288238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:20.288191    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn" event={"ID":"97632201-fdd8-4686-857e-04c4dcb8eddc","Type":"ContainerStarted","Data":"87d185f12801723ae0d7eed9603d278425a120425e36a87d4ecca4d8797629bc"}
Sep 15 09:27:20.305782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:20.305726    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn" podStartSLOduration=1.305711061 podStartE2EDuration="1.305711061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:27:20.303694067 +0000 UTC m=+1232.762983603" watchObservedRunningTime="2026-09-15 09:27:20.305711061 +0000 UTC m=+1232.765000601"
Sep 15 09:27:22.391062 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:22.391022    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"]
Sep 15 09:27:22.399998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:22.399961    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:27.026441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:27.026398    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:27.797344 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:27.797259    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:28.572967 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:28.572932    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:29.603110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:29.603068    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:29.603505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:29.603123    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:29.615525 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:29.615498    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:29.703473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:29.703438    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:30.069522 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:30.069439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:30.368082 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:30.368053    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:27:31.344247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:31.344214    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:31.593470 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:31.593428    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:32.296390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:32.296357    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:27:34.113739 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.113702    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:27:34.379094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.379006    2571 generic.go:356] "Generic (PLEG): container finished" podID="32f41605-709c-4a82-96f3-8b285971b171" containerID="3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619" exitCode=0
Sep 15 09:27:34.379094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.379050    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" event={"ID":"32f41605-709c-4a82-96f3-8b285971b171","Type":"ContainerDied","Data":"3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"}
Sep 15 09:27:34.379497 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.379481    2571 scope.go:117] "RemoveContainer" containerID="3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"
Sep 15 09:27:34.380651 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.380622    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:27:34.380733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.380685    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:27:34.380733 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:34.380713    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:27:35.387418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:35.387378    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" event={"ID":"32f41605-709c-4a82-96f3-8b285971b171","Type":"ContainerStarted","Data":"fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27"}
Sep 15 09:27:36.095387 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:36.095344    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:27:36.095574 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:36.095553    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 09:27:36.095635 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:36.095611    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 09:27:37.095309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.095263    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 09:27:37.095821 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.095336    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 09:27:37.412061 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.411961    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx" podUID="02e3c0ac-8430-41ab-a257-962ef793a3a5" containerName="main" containerID="cri-o://19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46" gracePeriod=45
Sep 15 09:27:37.702974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.702948    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:27:37.791129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791097    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-dshm\") pod \"02e3c0ac-8430-41ab-a257-962ef793a3a5\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") "
Sep 15 09:27:37.791129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791133    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e3c0ac-8430-41ab-a257-962ef793a3a5-tls-certs\") pod \"02e3c0ac-8430-41ab-a257-962ef793a3a5\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") "
Sep 15 09:27:37.791383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791180    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-tmp-dir\") pod \"02e3c0ac-8430-41ab-a257-962ef793a3a5\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") "
Sep 15 09:27:37.791383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7pbh\" (UniqueName: \"kubernetes.io/projected/02e3c0ac-8430-41ab-a257-962ef793a3a5-kube-api-access-w7pbh\") pod \"02e3c0ac-8430-41ab-a257-962ef793a3a5\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") "
Sep 15 09:27:37.791383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791270    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-model-cache\") pod \"02e3c0ac-8430-41ab-a257-962ef793a3a5\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") "
Sep 15 09:27:37.791383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791298    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-home\") pod \"02e3c0ac-8430-41ab-a257-962ef793a3a5\" (UID: \"02e3c0ac-8430-41ab-a257-962ef793a3a5\") "
Sep 15 09:27:37.791585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791462    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02e3c0ac-8430-41ab-a257-962ef793a3a5" (UID: "02e3c0ac-8430-41ab-a257-962ef793a3a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:27:37.791585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791478    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "02e3c0ac-8430-41ab-a257-962ef793a3a5" (UID: "02e3c0ac-8430-41ab-a257-962ef793a3a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:27:37.791585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791571    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-home" (OuterVolumeSpecName: "home") pod "02e3c0ac-8430-41ab-a257-962ef793a3a5" (UID: "02e3c0ac-8430-41ab-a257-962ef793a3a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:27:37.791742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791721    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:27:37.791783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791745    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:27:37.791783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.791761    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:27:37.793463 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.793437    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-dshm" (OuterVolumeSpecName: "dshm") pod "02e3c0ac-8430-41ab-a257-962ef793a3a5" (UID: "02e3c0ac-8430-41ab-a257-962ef793a3a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:27:37.793788 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.793767    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e3c0ac-8430-41ab-a257-962ef793a3a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02e3c0ac-8430-41ab-a257-962ef793a3a5" (UID: "02e3c0ac-8430-41ab-a257-962ef793a3a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:27:37.793890 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.793797    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e3c0ac-8430-41ab-a257-962ef793a3a5-kube-api-access-w7pbh" (OuterVolumeSpecName: "kube-api-access-w7pbh") pod "02e3c0ac-8430-41ab-a257-962ef793a3a5" (UID: "02e3c0ac-8430-41ab-a257-962ef793a3a5"). InnerVolumeSpecName "kube-api-access-w7pbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:27:37.892613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.892573    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e3c0ac-8430-41ab-a257-962ef793a3a5-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:27:37.892613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.892604    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e3c0ac-8430-41ab-a257-962ef793a3a5-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:27:37.892613 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:37.892615    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7pbh\" (UniqueName: \"kubernetes.io/projected/02e3c0ac-8430-41ab-a257-962ef793a3a5-kube-api-access-w7pbh\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:27:38.095999 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.095963    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 09:27:38.096397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.096019    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 09:27:38.403287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.403251    2571 generic.go:356] "Generic (PLEG): container finished" podID="02e3c0ac-8430-41ab-a257-962ef793a3a5" containerID="19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46" exitCode=0
Sep 15 09:27:38.403471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.403292    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"
Sep 15 09:27:38.403471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.403328    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx" event={"ID":"02e3c0ac-8430-41ab-a257-962ef793a3a5","Type":"ContainerDied","Data":"19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46"}
Sep 15 09:27:38.403471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.403370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx" event={"ID":"02e3c0ac-8430-41ab-a257-962ef793a3a5","Type":"ContainerDied","Data":"f2c4d5a325641f0b2f3a935a42cbd7530349c2c92eea633fd68706a9caa203b7"}
Sep 15 09:27:38.403471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.403386    2571 scope.go:117] "RemoveContainer" containerID="19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46"
Sep 15 09:27:38.412776 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.412637    2571 scope.go:117] "RemoveContainer" containerID="19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46"
Sep 15 09:27:38.412983 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:27:38.412965    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46\": container with ID starting with 19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46 not found: ID does not exist" containerID="19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46"
Sep 15 09:27:38.413045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.412995    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46"} err="failed to get container status \"19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46\": rpc error: code = NotFound desc = could not find container \"19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46\": container with ID starting with 19c95be23bb3fbb4fcc07b694b94c19392eda464efa91fdd7531ea3addb5de46 not found: ID does not exist"
Sep 15 09:27:38.443852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.439682    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"]
Sep 15 09:27:38.449856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:38.445708    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-b-fd935fd3-kserve-589fd655-qbznx"]
Sep 15 09:27:39.096149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:39.096114    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 09:27:39.096647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:39.096180    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 09:27:40.055973 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:40.055936    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e3c0ac-8430-41ab-a257-962ef793a3a5" path="/var/lib/kubelet/pods/02e3c0ac-8430-41ab-a257-962ef793a3a5/volumes"
Sep 15 09:27:40.096145 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:40.096073    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 15 09:27:40.096266 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:40.096135    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 15 09:27:41.099460 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:41.099423    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:27:41.099929 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:41.099705    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:27:41.100480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:41.100464    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:27:55.735986 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:55.735954    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"]
Sep 15 09:27:55.758543 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:27:55.758508    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:09.212088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.212051    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"]
Sep 15 09:28:09.212720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.212703    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e3c0ac-8430-41ab-a257-962ef793a3a5" containerName="main"
Sep 15 09:28:09.212777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.212724    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e3c0ac-8430-41ab-a257-962ef793a3a5" containerName="main"
Sep 15 09:28:09.212858 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.212845    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e3c0ac-8430-41ab-a257-962ef793a3a5" containerName="main"
Sep 15 09:28:09.218384 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.218361    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.222439 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.222412    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f79800\"/\"auth-test-model-f79800-kserve-self-signed-certs\""
Sep 15 09:28:09.222588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.222468    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f79800\"/\"default-dockercfg-f6pvl\""
Sep 15 09:28:09.222588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.222491    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f79800\"/\"kube-root-ca.crt\""
Sep 15 09:28:09.222840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.222692    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f79800\"/\"openshift-service-ca.crt\""
Sep 15 09:28:09.227501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.227462    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"]
Sep 15 09:28:09.251254 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.251219    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:09.328470 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.328431    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b47q\" (UniqueName: \"kubernetes.io/projected/0f260f8c-b7ab-4572-b603-5fdef8276330-kube-api-access-9b47q\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.328650 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.328537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f260f8c-b7ab-4572-b603-5fdef8276330-tls-certs\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.328650 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.328565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-home\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.328650 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.328590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-tmp-dir\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.328650 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.328619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-dshm\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.328791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.328729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-model-cache\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430254    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f260f8c-b7ab-4572-b603-5fdef8276330-tls-certs\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430503 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-home\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430503 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-tmp-dir\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430503 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430488    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-dshm\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-model-cache\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430614    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b47q\" (UniqueName: \"kubernetes.io/projected/0f260f8c-b7ab-4572-b603-5fdef8276330-kube-api-access-9b47q\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.430864 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.430839    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-home\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.431037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.431007    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-model-cache\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.431314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.431290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-tmp-dir\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.432910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.432878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-dshm\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.433221 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.433202    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f260f8c-b7ab-4572-b603-5fdef8276330-tls-certs\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.438841 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.438797    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b47q\" (UniqueName: \"kubernetes.io/projected/0f260f8c-b7ab-4572-b603-5fdef8276330-kube-api-access-9b47q\") pod \"auth-test-model-f79800-kserve-77b8b7b675-ljnxj\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") " pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.530736 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.530639    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:09.805258 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:09.805223    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"]
Sep 15 09:28:09.807203 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:28:09.807174    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f260f8c_b7ab_4572_b603_5fdef8276330.slice/crio-f11ad579aa23e6f7d4886ff69bc148ec80650f1b0d5b334cb2a216ff4d1f48cb WatchSource:0}: Error finding container f11ad579aa23e6f7d4886ff69bc148ec80650f1b0d5b334cb2a216ff4d1f48cb: Status 404 returned error can't find the container with id f11ad579aa23e6f7d4886ff69bc148ec80650f1b0d5b334cb2a216ff4d1f48cb
Sep 15 09:28:10.583544 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:10.583505    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj" event={"ID":"0f260f8c-b7ab-4572-b603-5fdef8276330","Type":"ContainerStarted","Data":"53183c5287879c310f7a1f44c546e8aa6d2a64068a14ff7c825970b73463ff20"}
Sep 15 09:28:10.583954 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:10.583555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj" event={"ID":"0f260f8c-b7ab-4572-b603-5fdef8276330","Type":"ContainerStarted","Data":"f11ad579aa23e6f7d4886ff69bc148ec80650f1b0d5b334cb2a216ff4d1f48cb"}
Sep 15 09:28:10.600487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:10.600433    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj" podStartSLOduration=1.600415575 podStartE2EDuration="1.600415575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:10.599681895 +0000 UTC m=+1283.058971438" watchObservedRunningTime="2026-09-15 09:28:10.600415575 +0000 UTC m=+1283.059705157"
Sep 15 09:28:10.755394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:10.755351    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92" podUID="5e2cf491-3d55-4181-88a8-98d42126ecd1" containerName="main" containerID="cri-o://d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748" gracePeriod=45
Sep 15 09:28:11.014745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.014718    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:28:11.149213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149120    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-dshm\") pod \"5e2cf491-3d55-4181-88a8-98d42126ecd1\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") "
Sep 15 09:28:11.149213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149162    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-home\") pod \"5e2cf491-3d55-4181-88a8-98d42126ecd1\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") "
Sep 15 09:28:11.149430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149229    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-tmp-dir\") pod \"5e2cf491-3d55-4181-88a8-98d42126ecd1\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") "
Sep 15 09:28:11.149430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149262    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbzkd\" (UniqueName: \"kubernetes.io/projected/5e2cf491-3d55-4181-88a8-98d42126ecd1-kube-api-access-mbzkd\") pod \"5e2cf491-3d55-4181-88a8-98d42126ecd1\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") "
Sep 15 09:28:11.149430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149309    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-model-cache\") pod \"5e2cf491-3d55-4181-88a8-98d42126ecd1\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") "
Sep 15 09:28:11.149430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149364    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2cf491-3d55-4181-88a8-98d42126ecd1-tls-certs\") pod \"5e2cf491-3d55-4181-88a8-98d42126ecd1\" (UID: \"5e2cf491-3d55-4181-88a8-98d42126ecd1\") "
Sep 15 09:28:11.149610 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149541    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e2cf491-3d55-4181-88a8-98d42126ecd1" (UID: "5e2cf491-3d55-4181-88a8-98d42126ecd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:28:11.149610 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149536    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-home" (OuterVolumeSpecName: "home") pod "5e2cf491-3d55-4181-88a8-98d42126ecd1" (UID: "5e2cf491-3d55-4181-88a8-98d42126ecd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:28:11.149740 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149572    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e2cf491-3d55-4181-88a8-98d42126ecd1" (UID: "5e2cf491-3d55-4181-88a8-98d42126ecd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:28:11.149851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149813    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:28:11.149917 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149857    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:28:11.149917 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.149869    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:28:11.151485 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.151462    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-dshm" (OuterVolumeSpecName: "dshm") pod "5e2cf491-3d55-4181-88a8-98d42126ecd1" (UID: "5e2cf491-3d55-4181-88a8-98d42126ecd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:28:11.151581 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.151490    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2cf491-3d55-4181-88a8-98d42126ecd1-kube-api-access-mbzkd" (OuterVolumeSpecName: "kube-api-access-mbzkd") pod "5e2cf491-3d55-4181-88a8-98d42126ecd1" (UID: "5e2cf491-3d55-4181-88a8-98d42126ecd1"). InnerVolumeSpecName "kube-api-access-mbzkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:28:11.151581 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.151510    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2cf491-3d55-4181-88a8-98d42126ecd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e2cf491-3d55-4181-88a8-98d42126ecd1" (UID: "5e2cf491-3d55-4181-88a8-98d42126ecd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:28:11.250524 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.250475    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2cf491-3d55-4181-88a8-98d42126ecd1-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:28:11.250695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.250544    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbzkd\" (UniqueName: \"kubernetes.io/projected/5e2cf491-3d55-4181-88a8-98d42126ecd1-kube-api-access-mbzkd\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:28:11.250695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.250558    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2cf491-3d55-4181-88a8-98d42126ecd1-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:28:11.590572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.590540    2571 generic.go:356] "Generic (PLEG): container finished" podID="5e2cf491-3d55-4181-88a8-98d42126ecd1" containerID="d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748" exitCode=0
Sep 15 09:28:11.591053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.590597    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"
Sep 15 09:28:11.591053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.590622    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92" event={"ID":"5e2cf491-3d55-4181-88a8-98d42126ecd1","Type":"ContainerDied","Data":"d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748"}
Sep 15 09:28:11.591053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.590661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92" event={"ID":"5e2cf491-3d55-4181-88a8-98d42126ecd1","Type":"ContainerDied","Data":"c115d906ede9787e9b58ba96ffa26021801cbf514ac7201f06b89ac687d9d0c0"}
Sep 15 09:28:11.591053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.590678    2571 scope.go:117] "RemoveContainer" containerID="d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748"
Sep 15 09:28:11.600016 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.599999    2571 scope.go:117] "RemoveContainer" containerID="d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748"
Sep 15 09:28:11.600318 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:28:11.600299    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748\": container with ID starting with d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748 not found: ID does not exist" containerID="d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748"
Sep 15 09:28:11.600383 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.600328    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748"} err="failed to get container status \"d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748\": rpc error: code = NotFound desc = could not find container \"d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748\": container with ID starting with d723926bf734b2efc4a739cc1d23261ed98926753bedc7e0a7306a5d23586748 not found: ID does not exist"
Sep 15 09:28:11.611732 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.611702    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"]
Sep 15 09:28:11.616427 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:11.616402    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/e2e-conflict-a-fd935fd3-kserve-666ddb7b97-stl92"]
Sep 15 09:28:12.056234 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:12.056151    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2cf491-3d55-4181-88a8-98d42126ecd1" path="/var/lib/kubelet/pods/5e2cf491-3d55-4181-88a8-98d42126ecd1/volumes"
Sep 15 09:28:17.220675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:17.220636    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:28:18.907365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.907323    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"]
Sep 15 09:28:18.907751 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.907737    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2cf491-3d55-4181-88a8-98d42126ecd1" containerName="main"
Sep 15 09:28:18.907796 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.907752    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2cf491-3d55-4181-88a8-98d42126ecd1" containerName="main"
Sep 15 09:28:18.907865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.907855    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2cf491-3d55-4181-88a8-98d42126ecd1" containerName="main"
Sep 15 09:28:18.911049 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.911030    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:18.913446 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.913414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"distinct-w4-890f3e-kserve-self-signed-certs\""
Sep 15 09:28:18.919714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.919689    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"]
Sep 15 09:28:18.931140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:18.931108    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:19.036451 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.036411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-home\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.036636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.036467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8xp\" (UniqueName: \"kubernetes.io/projected/dd9b79dd-3444-4992-b16a-c68f9c608687-kube-api-access-zx8xp\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.036636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.036591    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-tmp-dir\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.036636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.036618    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b79dd-3444-4992-b16a-c68f9c608687-tls-certs\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.036748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.036677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-dshm\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.036748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.036701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-model-cache\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.137468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-home\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.137687 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8xp\" (UniqueName: \"kubernetes.io/projected/dd9b79dd-3444-4992-b16a-c68f9c608687-kube-api-access-zx8xp\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.137687 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137520    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-tmp-dir\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.137687 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137538    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b79dd-3444-4992-b16a-c68f9c608687-tls-certs\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.137687 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-dshm\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.137687 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-model-cache\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.138002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.137971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-home\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.138056 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.138012    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-tmp-dir\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.138102 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.138051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-model-cache\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.140056 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.140033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-dshm\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.140308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.140286    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b79dd-3444-4992-b16a-c68f9c608687-tls-certs\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.146903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.146877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8xp\" (UniqueName: \"kubernetes.io/projected/dd9b79dd-3444-4992-b16a-c68f9c608687-kube-api-access-zx8xp\") pod \"distinct-w4-890f3e-kserve-785755c694-ktc4b\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.221474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.221379    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:19.497034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.497007    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"]
Sep 15 09:28:19.531493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.531464    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:19.531593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.531515    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:19.544689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.544650    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:19.637535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.637500    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b" event={"ID":"dd9b79dd-3444-4992-b16a-c68f9c608687","Type":"ContainerStarted","Data":"2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3"}
Sep 15 09:28:19.637715 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.637544    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b" event={"ID":"dd9b79dd-3444-4992-b16a-c68f9c608687","Type":"ContainerStarted","Data":"31b9898858c6476b9cf95deea5fe030b710b6e558ed77c3fb59351f0a0bc258a"}
Sep 15 09:28:19.648698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.648670    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:28:19.655164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:19.655119    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b" podStartSLOduration=1.655104715 podStartE2EDuration="1.655104715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:28:19.652699929 +0000 UTC m=+1292.111989482" watchObservedRunningTime="2026-09-15 09:28:19.655104715 +0000 UTC m=+1292.114394294"
Sep 15 09:28:23.699377 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:23.699343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:24.523151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:24.523117    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:24.623029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:24.622992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:25.371457 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:25.371423    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:26.168781 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:26.168743    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:26.976609 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:26.976568    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:27.768469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:27.768378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:28.434120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:28.434084    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:29.222470 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:29.222435    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:29.222650 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:29.222489    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:29.235558 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:29.235530    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:29.275089 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:29.275030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:29.701594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:29.701560    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:28:30.104274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:30.104229    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:39.141163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:39.141133    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:39.213033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:39.213007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:39.313923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:39.313892    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:39.413538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:39.413448    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:39.762572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:39.762491    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:40.603930 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:40.603892    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:41.290820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:41.290782    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:42.145418 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:42.145387    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:42.784324 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:42.784284    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:42.884770 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:42.884738    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:43.489496 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:43.489457    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:44.339757 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:44.339692    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:47.762560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:47.762515    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"]
Sep 15 09:28:47.858152 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:47.858122    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:28:51.008921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:28:51.008881    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:29:02.782923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:02.782874    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn" podUID="97632201-fdd8-4686-857e-04c4dcb8eddc" containerName="main" containerID="cri-o://b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1" gracePeriod=45
Sep 15 09:29:03.045409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.045344    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:29:03.120745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.120706    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-model-cache\") pod \"97632201-fdd8-4686-857e-04c4dcb8eddc\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") "
Sep 15 09:29:03.120921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.120769    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86jpm\" (UniqueName: \"kubernetes.io/projected/97632201-fdd8-4686-857e-04c4dcb8eddc-kube-api-access-86jpm\") pod \"97632201-fdd8-4686-857e-04c4dcb8eddc\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") "
Sep 15 09:29:03.120921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.120806    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97632201-fdd8-4686-857e-04c4dcb8eddc-tls-certs\") pod \"97632201-fdd8-4686-857e-04c4dcb8eddc\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") "
Sep 15 09:29:03.120921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.120880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-tmp-dir\") pod \"97632201-fdd8-4686-857e-04c4dcb8eddc\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") "
Sep 15 09:29:03.120921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.120905    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-dshm\") pod \"97632201-fdd8-4686-857e-04c4dcb8eddc\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") "
Sep 15 09:29:03.121146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.120930    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-home\") pod \"97632201-fdd8-4686-857e-04c4dcb8eddc\" (UID: \"97632201-fdd8-4686-857e-04c4dcb8eddc\") "
Sep 15 09:29:03.121146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.121013    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-model-cache" (OuterVolumeSpecName: "model-cache") pod "97632201-fdd8-4686-857e-04c4dcb8eddc" (UID: "97632201-fdd8-4686-857e-04c4dcb8eddc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:29:03.121285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.121259    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-home" (OuterVolumeSpecName: "home") pod "97632201-fdd8-4686-857e-04c4dcb8eddc" (UID: "97632201-fdd8-4686-857e-04c4dcb8eddc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:29:03.121412 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.121284    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97632201-fdd8-4686-857e-04c4dcb8eddc" (UID: "97632201-fdd8-4686-857e-04c4dcb8eddc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:29:03.121571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.121555    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:29:03.121639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.121576    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:29:03.121639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.121589    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:29:03.123271 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.123241    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97632201-fdd8-4686-857e-04c4dcb8eddc-kube-api-access-86jpm" (OuterVolumeSpecName: "kube-api-access-86jpm") pod "97632201-fdd8-4686-857e-04c4dcb8eddc" (UID: "97632201-fdd8-4686-857e-04c4dcb8eddc"). InnerVolumeSpecName "kube-api-access-86jpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:29:03.123491 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.123475    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97632201-fdd8-4686-857e-04c4dcb8eddc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97632201-fdd8-4686-857e-04c4dcb8eddc" (UID: "97632201-fdd8-4686-857e-04c4dcb8eddc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:29:03.123559 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.123504    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-dshm" (OuterVolumeSpecName: "dshm") pod "97632201-fdd8-4686-857e-04c4dcb8eddc" (UID: "97632201-fdd8-4686-857e-04c4dcb8eddc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:29:03.223242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.223206    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86jpm\" (UniqueName: \"kubernetes.io/projected/97632201-fdd8-4686-857e-04c4dcb8eddc-kube-api-access-86jpm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:29:03.223242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.223237    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97632201-fdd8-4686-857e-04c4dcb8eddc-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:29:03.223242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.223250    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97632201-fdd8-4686-857e-04c4dcb8eddc-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:29:03.875819 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.875780    2571 generic.go:356] "Generic (PLEG): container finished" podID="97632201-fdd8-4686-857e-04c4dcb8eddc" containerID="b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1" exitCode=0
Sep 15 09:29:03.876289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.875866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn" event={"ID":"97632201-fdd8-4686-857e-04c4dcb8eddc","Type":"ContainerDied","Data":"b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1"}
Sep 15 09:29:03.876289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.875911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn" event={"ID":"97632201-fdd8-4686-857e-04c4dcb8eddc","Type":"ContainerDied","Data":"87d185f12801723ae0d7eed9603d278425a120425e36a87d4ecca4d8797629bc"}
Sep 15 09:29:03.876289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.875933    2571 scope.go:117] "RemoveContainer" containerID="b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1"
Sep 15 09:29:03.876289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.875876    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"
Sep 15 09:29:03.893289 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.893264    2571 scope.go:117] "RemoveContainer" containerID="b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1"
Sep 15 09:29:03.893648 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:29:03.893626    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1\": container with ID starting with b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1 not found: ID does not exist" containerID="b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1"
Sep 15 09:29:03.893726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.893657    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1"} err="failed to get container status \"b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1\": rpc error: code = NotFound desc = could not find container \"b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1\": container with ID starting with b5481bf032476062ca6ed1999d77a48e7ec21e58401e5ecd9fed174f06911fc1 not found: ID does not exist"
Sep 15 09:29:03.902542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.902513    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"]
Sep 15 09:29:03.904523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:03.904501    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fcb750c6/e2e-lifecycle-model-fcb750c6-kserve-7854766df9-vj9kn"]
Sep 15 09:29:04.056003 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:04.055970    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97632201-fdd8-4686-857e-04c4dcb8eddc" path="/var/lib/kubelet/pods/97632201-fdd8-4686-857e-04c4dcb8eddc/volumes"
Sep 15 09:29:22.606509 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:22.606470    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:29:24.486504 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:24.486475    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:29:24.837320 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:24.837289    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:29:30.369730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:30.369695    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:29:50.546995 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:50.546963    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:29:51.908657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:29:51.908619    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:30:00.147848 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.147792    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"]
Sep 15 09:30:00.148334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.148248    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97632201-fdd8-4686-857e-04c4dcb8eddc" containerName="main"
Sep 15 09:30:00.148334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.148261    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="97632201-fdd8-4686-857e-04c4dcb8eddc" containerName="main"
Sep 15 09:30:00.148409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.148346    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="97632201-fdd8-4686-857e-04c4dcb8eddc" containerName="main"
Sep 15 09:30:00.152569 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.152549    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"]
Sep 15 09:30:00.152675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.152654    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"
Sep 15 09:30:00.155565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.155539    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b4dqq\""
Sep 15 09:30:00.156278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.156235    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"
Sep 15 09:30:00.163608 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.163573    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"]
Sep 15 09:30:00.167127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.167097    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"]
Sep 15 09:30:00.167752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.167729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"
Sep 15 09:30:00.170884 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.170850    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"]
Sep 15 09:30:00.175425 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.175402    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"
Sep 15 09:30:00.176937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.176920    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"
Sep 15 09:30:00.182277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.182109    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"]
Sep 15 09:30:00.215695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.215654    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"
Sep 15 09:30:00.460691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.460664    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"]
Sep 15 09:30:00.462794 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:30:00.462763    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b397415_b7d9_4d02_ad7a_a4ac043988cc.slice/crio-2037feccb78d6d5d772e86f7d66c7cc53fd3e53ea50a2fb115830d9827edcc0f WatchSource:0}: Error finding container 2037feccb78d6d5d772e86f7d66c7cc53fd3e53ea50a2fb115830d9827edcc0f: Status 404 returned error can't find the container with id 2037feccb78d6d5d772e86f7d66c7cc53fd3e53ea50a2fb115830d9827edcc0f
Sep 15 09:30:00.472779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.472757    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"]
Sep 15 09:30:00.474887 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:30:00.474860    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f262fc_babb_4f5e_8e94_5156f738cc3e.slice/crio-5fa77323db91a026280005704cf9abee20911579ff7b17d881b84aaef3dcbcb0 WatchSource:0}: Error finding container 5fa77323db91a026280005704cf9abee20911579ff7b17d881b84aaef3dcbcb0: Status 404 returned error can't find the container with id 5fa77323db91a026280005704cf9abee20911579ff7b17d881b84aaef3dcbcb0
Sep 15 09:30:00.509653 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:00.509627    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"]
Sep 15 09:30:00.512031 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:30:00.511998    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e15600_83ee_4412_93d6_e7f12178d6c5.slice/crio-0c4b174baea1b53c45b02ae0a8148755de5971c59a698d9f2f332f315dc3e410 WatchSource:0}: Error finding container 0c4b174baea1b53c45b02ae0a8148755de5971c59a698d9f2f332f315dc3e410: Status 404 returned error can't find the container with id 0c4b174baea1b53c45b02ae0a8148755de5971c59a698d9f2f332f315dc3e410
Sep 15 09:30:01.200589 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:01.200556    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t" event={"ID":"0b397415-b7d9-4d02-ad7a-a4ac043988cc","Type":"ContainerStarted","Data":"2037feccb78d6d5d772e86f7d66c7cc53fd3e53ea50a2fb115830d9827edcc0f"}
Sep 15 09:30:01.201698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:01.201674    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx" event={"ID":"94f262fc-babb-4f5e-8e94-5156f738cc3e","Type":"ContainerStarted","Data":"5fa77323db91a026280005704cf9abee20911579ff7b17d881b84aaef3dcbcb0"}
Sep 15 09:30:01.202869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:01.202847    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr" event={"ID":"02e15600-83ee-4412-93d6-e7f12178d6c5","Type":"ContainerStarted","Data":"0c4b174baea1b53c45b02ae0a8148755de5971c59a698d9f2f332f315dc3e410"}
Sep 15 09:30:02.023665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:02.023630    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:02.124755 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:02.124678    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:02.418369 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:02.418258    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:03.468686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:03.468647    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:03.568983 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:03.568946    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:03.631657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:03.631572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:04.235758 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:04.235719    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b397415-b7d9-4d02-ad7a-a4ac043988cc" containerID="9e4fb8a446310bf02869cde3a9c4db35911e20a58e596d7f934a48ae0ae6be52" exitCode=0
Sep 15 09:30:04.235906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:04.235876    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t" event={"ID":"0b397415-b7d9-4d02-ad7a-a4ac043988cc","Type":"ContainerDied","Data":"9e4fb8a446310bf02869cde3a9c4db35911e20a58e596d7f934a48ae0ae6be52"}
Sep 15 09:30:04.240191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:04.239251    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx" event={"ID":"94f262fc-babb-4f5e-8e94-5156f738cc3e","Type":"ContainerStarted","Data":"a49fd3276ef5eb091fa8b6c8514f3a34a1ed6a0aec6ac5dd82f995f06efb2992"}
Sep 15 09:30:04.241412 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:04.241229    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr" event={"ID":"02e15600-83ee-4412-93d6-e7f12178d6c5","Type":"ContainerStarted","Data":"0ff415d8b92cc450397116623a2f40f31a5e98b60853cd07418f1babaf337007"}
Sep 15 09:30:04.926290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:04.926254    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:05.027591 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.027558    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:05.250674 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.250580    2571 generic.go:356] "Generic (PLEG): container finished" podID="02e15600-83ee-4412-93d6-e7f12178d6c5" containerID="0ff415d8b92cc450397116623a2f40f31a5e98b60853cd07418f1babaf337007" exitCode=0
Sep 15 09:30:05.250987 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.250616    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr" event={"ID":"02e15600-83ee-4412-93d6-e7f12178d6c5","Type":"ContainerDied","Data":"0ff415d8b92cc450397116623a2f40f31a5e98b60853cd07418f1babaf337007"}
Sep 15 09:30:05.254229 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.254197    2571 generic.go:356] "Generic (PLEG): container finished" podID="94f262fc-babb-4f5e-8e94-5156f738cc3e" containerID="a49fd3276ef5eb091fa8b6c8514f3a34a1ed6a0aec6ac5dd82f995f06efb2992" exitCode=0
Sep 15 09:30:05.254396 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.254310    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx" event={"ID":"94f262fc-babb-4f5e-8e94-5156f738cc3e","Type":"ContainerDied","Data":"a49fd3276ef5eb091fa8b6c8514f3a34a1ed6a0aec6ac5dd82f995f06efb2992"}
Sep 15 09:30:05.438517 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.438473    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"
Sep 15 09:30:05.442565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.442460    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"
Sep 15 09:30:05.470421 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.470385    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"
Sep 15 09:30:05.788343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:05.788312    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:06.261547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.261519    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"
Sep 15 09:30:06.262085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.261558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr" event={"ID":"02e15600-83ee-4412-93d6-e7f12178d6c5","Type":"ContainerDied","Data":"0c4b174baea1b53c45b02ae0a8148755de5971c59a698d9f2f332f315dc3e410"}
Sep 15 09:30:06.262085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.261608    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4b174baea1b53c45b02ae0a8148755de5971c59a698d9f2f332f315dc3e410"
Sep 15 09:30:06.263023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.263002    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"
Sep 15 09:30:06.263146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.263033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t" event={"ID":"0b397415-b7d9-4d02-ad7a-a4ac043988cc","Type":"ContainerDied","Data":"2037feccb78d6d5d772e86f7d66c7cc53fd3e53ea50a2fb115830d9827edcc0f"}
Sep 15 09:30:06.263146 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.263077    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2037feccb78d6d5d772e86f7d66c7cc53fd3e53ea50a2fb115830d9827edcc0f"
Sep 15 09:30:06.264645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.264619    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"
Sep 15 09:30:06.264765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.264640    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx" event={"ID":"94f262fc-babb-4f5e-8e94-5156f738cc3e","Type":"ContainerDied","Data":"5fa77323db91a026280005704cf9abee20911579ff7b17d881b84aaef3dcbcb0"}
Sep 15 09:30:06.264765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:06.264688    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa77323db91a026280005704cf9abee20911579ff7b17d881b84aaef3dcbcb0"
Sep 15 09:30:11.524295 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:11.524247    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:11.624333 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:11.624289    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:12.214051 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:12.214020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:12.391794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:12.391749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:14.414432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414395    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"]
Sep 15 09:30:14.414852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414814    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f262fc-babb-4f5e-8e94-5156f738cc3e" containerName="cleanup"
Sep 15 09:30:14.414852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414843    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f262fc-babb-4f5e-8e94-5156f738cc3e" containerName="cleanup"
Sep 15 09:30:14.414937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414867    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e15600-83ee-4412-93d6-e7f12178d6c5" containerName="cleanup"
Sep 15 09:30:14.414937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414873    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e15600-83ee-4412-93d6-e7f12178d6c5" containerName="cleanup"
Sep 15 09:30:14.414937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414883    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b397415-b7d9-4d02-ad7a-a4ac043988cc" containerName="cleanup"
Sep 15 09:30:14.414937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414889    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b397415-b7d9-4d02-ad7a-a4ac043988cc" containerName="cleanup"
Sep 15 09:30:14.415079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414967    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b397415-b7d9-4d02-ad7a-a4ac043988cc" containerName="cleanup"
Sep 15 09:30:14.415079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414979    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f262fc-babb-4f5e-8e94-5156f738cc3e" containerName="cleanup"
Sep 15 09:30:14.415079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.414989    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e15600-83ee-4412-93d6-e7f12178d6c5" containerName="cleanup"
Sep 15 09:30:14.418043 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.418024    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.423660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.423627    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-890f3e\"/\"distinct-2-w4-890f3e-kserve-self-signed-certs\""
Sep 15 09:30:14.432688 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.432661    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"]
Sep 15 09:30:14.438772 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.438746    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:14.513727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.513688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-home\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.513727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.513727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-dshm\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.514035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.513749    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6pj\" (UniqueName: \"kubernetes.io/projected/143d6821-6aff-4ab0-9e07-0b52eec93279-kube-api-access-pk6pj\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.514035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.513797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-tmp-dir\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.514035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.513882    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-model-cache\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.514035 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.513982    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143d6821-6aff-4ab0-9e07-0b52eec93279-tls-certs\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.615579 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.615538    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-dshm\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.615579 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.615595    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6pj\" (UniqueName: \"kubernetes.io/projected/143d6821-6aff-4ab0-9e07-0b52eec93279-kube-api-access-pk6pj\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.615869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.615663    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-tmp-dir\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.615869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.615721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-model-cache\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.615869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.615795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143d6821-6aff-4ab0-9e07-0b52eec93279-tls-certs\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.616053 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.615961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-home\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.616169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.616147    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-tmp-dir\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.616236 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.616183    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-model-cache\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.616278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.616253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-home\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.618472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.618448    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-dshm\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.618685 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.618669    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143d6821-6aff-4ab0-9e07-0b52eec93279-tls-certs\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.624203 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.624179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6pj\" (UniqueName: \"kubernetes.io/projected/143d6821-6aff-4ab0-9e07-0b52eec93279-kube-api-access-pk6pj\") pod \"distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") " pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.728304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.728213    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:14.811151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.811096    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:14.863563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:14.863526    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:15.002535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:15.002507    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"]
Sep 15 09:30:15.004003 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:30:15.003975    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod143d6821_6aff_4ab0_9e07_0b52eec93279.slice/crio-6297a6a3282a69ab06efa53e051260bcc2f955f7e5e79a229e544ec7332266ee WatchSource:0}: Error finding container 6297a6a3282a69ab06efa53e051260bcc2f955f7e5e79a229e544ec7332266ee: Status 404 returned error can't find the container with id 6297a6a3282a69ab06efa53e051260bcc2f955f7e5e79a229e544ec7332266ee
Sep 15 09:30:15.313180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:15.313088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf" event={"ID":"143d6821-6aff-4ab0-9e07-0b52eec93279","Type":"ContainerStarted","Data":"c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153"}
Sep 15 09:30:15.313180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:15.313133    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf" event={"ID":"143d6821-6aff-4ab0-9e07-0b52eec93279","Type":"ContainerStarted","Data":"6297a6a3282a69ab06efa53e051260bcc2f955f7e5e79a229e544ec7332266ee"}
Sep 15 09:30:15.332004 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:15.331953    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf" podStartSLOduration=1.33193733 podStartE2EDuration="1.33193733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:30:15.330215939 +0000 UTC m=+1407.789505503" watchObservedRunningTime="2026-09-15 09:30:15.33193733 +0000 UTC m=+1407.791226871"
Sep 15 09:30:16.307816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:16.307770    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:30:17.627778 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:17.627742    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:18.086378 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:18.086343    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:18.151529 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:18.151494    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:18.230018 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:18.229981    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:18.326262 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:18.326222    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:20.147002 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:20.146966    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:24.729400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:24.729364    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:24.729812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:24.729421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:24.742222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:24.742194    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:25.379415 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:25.379381    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:30:28.781408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:28.781372    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:29.004538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:29.004498    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:30.934253 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:30.934199    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:43.950167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:43.950058    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:30:47.760094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:47.760057    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:49.920858 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:49.920797    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:55.810282 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:55.810250    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:55.865557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:55.865524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:56.135424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:56.135386    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:56.438417 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:56.438338    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:56.605078 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:56.605041    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:59.368313 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:59.368276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:59.661065 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:59.660985    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:30:59.785896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:30:59.785864    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:00.719536 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:00.719500    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:01.596042 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:01.596010    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:02.007744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:02.007654    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:02.896458 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:02.896425    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:03.573318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:03.573282    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:04.270501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:04.270466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:04.370761 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:04.370732    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:04.571074 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:04.570990    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:04.772852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:04.772797    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:04.877411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:04.877374    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:20.209956 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:20.209917    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:31:26.618232 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:26.618193    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:27.442044 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:27.442006    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:28.124088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:28.124023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:28.810512 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:28.810476    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:29.574052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:29.574016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:29.732936 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:29.732902    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:30.188665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.188629    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:30.291444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.291414    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:30.475250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.475167    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"]
Sep 15 09:31:30.485286 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.485256    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.488420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.488394    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ff390e-openshift-default-dockercfg-jvmqr\""
Sep 15 09:31:30.491121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.491095    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"]
Sep 15 09:31:30.508107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.508075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:30.659560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-data\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62sd\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-kube-api-access-p62sd\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-podinfo\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659728    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-socket\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istiod-ca-cert\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-certs\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-credential-socket\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-token\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.659992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.659906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-envoy\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761052    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62sd\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-kube-api-access-p62sd\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-podinfo\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761149 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-socket\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istiod-ca-cert\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-certs\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-credential-socket\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-token\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761294    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-envoy\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-data\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761606    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-socket\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761630    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-certs\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761704    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-credential-socket\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.761922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.761767    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-data\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.762134 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.762114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istiod-ca-cert\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.763620 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.763599    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-envoy\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.763765 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.763748    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-podinfo\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.768792 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.768767    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-token\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.769094 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.769075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62sd\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-kube-api-access-p62sd\") pod \"e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") " pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:30.799133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:30.799097    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:31.035871 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.035759    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:31.079173 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.079143    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"]
Sep 15 09:31:31.081281 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:31:31.081257    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbce716_d8fe_4bb3_93eb_546af82f8b84.slice/crio-bc33440f9b1097a2201aef2dc43293afff34f53846b891bc409da94eae3d112b WatchSource:0}: Error finding container bc33440f9b1097a2201aef2dc43293afff34f53846b891bc409da94eae3d112b: Status 404 returned error can't find the container with id bc33440f9b1097a2201aef2dc43293afff34f53846b891bc409da94eae3d112b
Sep 15 09:31:31.083817 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.083784    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:31:31.083955 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.083907    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:31:31.083955 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.083940    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:31:31.303373 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.303286    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:31.742792 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.742756    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" event={"ID":"ddbce716-d8fe-4bb3-93eb-546af82f8b84","Type":"ContainerStarted","Data":"829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"}
Sep 15 09:31:31.743222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.742809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" event={"ID":"ddbce716-d8fe-4bb3-93eb-546af82f8b84","Type":"ContainerStarted","Data":"bc33440f9b1097a2201aef2dc43293afff34f53846b891bc409da94eae3d112b"}
Sep 15 09:31:31.768426 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.768371    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podStartSLOduration=1.7683559340000001 podStartE2EDuration="1.768355934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:31:31.765208264 +0000 UTC m=+1484.224497813" watchObservedRunningTime="2026-09-15 09:31:31.768355934 +0000 UTC m=+1484.227645474"
Sep 15 09:31:31.799423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:31.799388    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:32.018894 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:32.018793    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:32.805549 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:32.805517    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:33.019436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:33.019403    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:33.612455 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:33.612422    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:33.710214 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:33.710180    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:33.753339 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:33.753309    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:33.754309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:33.754284    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:31:34.015292 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:34.015205    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:34.780937 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:34.780901    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:35.627551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:35.627517    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:36.415642 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:36.415601    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:37.274107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:37.274075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:38.058588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:38.058554    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:38.757196 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:38.757156    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:39.963256 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:39.963215    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:40.064196 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:40.064164    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:40.146411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:40.146378    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:43.136702 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:43.136664    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:43.591438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:43.591401    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:44.465905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:44.465859    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:45.131551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:45.131520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:45.834336 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:45.834292    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:46.514692 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:46.514660    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:47.303116 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:47.303080    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:47.973665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:47.973636    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:48.084079 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:48.084052    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:31:48.088605 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:48.088580    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:31:48.120844 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:48.120800    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"]
Sep 15 09:31:48.149260 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:48.149224    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"]
Sep 15 09:31:48.160481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:48.160447    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"]
Sep 15 09:31:48.208260 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:48.208227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:51.572798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:51.572762    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:51.673212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:51.673177    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:52.041551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:52.041520    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:52.733055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:52.733019    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:53.423809 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:53.423771    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:53.523858 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:53.523805    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:54.206730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:54.206694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:55.088115 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:55.088075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:55.777796 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:55.777765    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:56.749976 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:56.749945    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:57.475965 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:57.475935    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:58.386973 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:58.386929    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:59.066535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:59.066501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:31:59.761176 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:31:59.761142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:00.581860 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:00.581813    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:01.272021 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:01.271988    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:01.661670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:01.661636    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:32:02.232674 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:02.232641    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:03.076560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.076524    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:03.142189 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.142123    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv" podUID="ea195e13-0b07-427d-9183-8c44672ed1fc" containerName="main" containerID="cri-o://3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec" gracePeriod=45
Sep 15 09:32:03.170851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.170760    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv" podUID="a434ac87-3ccc-4156-b0e2-2b53398137c9" containerName="main" containerID="cri-o://2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d" gracePeriod=45
Sep 15 09:32:03.181390 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.181342    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57" podUID="cdc38895-56fe-41cb-b5d8-5df27fe97798" containerName="main" containerID="cri-o://c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734" gracePeriod=45
Sep 15 09:32:03.516066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.516036    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:32:03.537861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.537389    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea195e13-0b07-427d-9183-8c44672ed1fc-tls-certs\") pod \"ea195e13-0b07-427d-9183-8c44672ed1fc\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") "
Sep 15 09:32:03.537861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.537433    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-tmp-dir\") pod \"ea195e13-0b07-427d-9183-8c44672ed1fc\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") "
Sep 15 09:32:03.537861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.537471    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-dshm\") pod \"ea195e13-0b07-427d-9183-8c44672ed1fc\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") "
Sep 15 09:32:03.537861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.537502    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-home\") pod \"ea195e13-0b07-427d-9183-8c44672ed1fc\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") "
Sep 15 09:32:03.537861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.537594    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-model-cache\") pod \"ea195e13-0b07-427d-9183-8c44672ed1fc\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") "
Sep 15 09:32:03.537861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.537650    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdtmw\" (UniqueName: \"kubernetes.io/projected/ea195e13-0b07-427d-9183-8c44672ed1fc-kube-api-access-hdtmw\") pod \"ea195e13-0b07-427d-9183-8c44672ed1fc\" (UID: \"ea195e13-0b07-427d-9183-8c44672ed1fc\") "
Sep 15 09:32:03.538510 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.538342    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-home" (OuterVolumeSpecName: "home") pod "ea195e13-0b07-427d-9183-8c44672ed1fc" (UID: "ea195e13-0b07-427d-9183-8c44672ed1fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.538728 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.538572    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea195e13-0b07-427d-9183-8c44672ed1fc" (UID: "ea195e13-0b07-427d-9183-8c44672ed1fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.540011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.539358    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.540011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.539384    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.542645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.541211    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea195e13-0b07-427d-9183-8c44672ed1fc" (UID: "ea195e13-0b07-427d-9183-8c44672ed1fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.542645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.541491    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-dshm" (OuterVolumeSpecName: "dshm") pod "ea195e13-0b07-427d-9183-8c44672ed1fc" (UID: "ea195e13-0b07-427d-9183-8c44672ed1fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.545284 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.545253    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea195e13-0b07-427d-9183-8c44672ed1fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea195e13-0b07-427d-9183-8c44672ed1fc" (UID: "ea195e13-0b07-427d-9183-8c44672ed1fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:32:03.547580 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.547552    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea195e13-0b07-427d-9183-8c44672ed1fc-kube-api-access-hdtmw" (OuterVolumeSpecName: "kube-api-access-hdtmw") pod "ea195e13-0b07-427d-9183-8c44672ed1fc" (UID: "ea195e13-0b07-427d-9183-8c44672ed1fc"). InnerVolumeSpecName "kube-api-access-hdtmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:32:03.553642 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.553619    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:32:03.557569 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.557547    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:32:03.640584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640545    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-home\") pod \"a434ac87-3ccc-4156-b0e2-2b53398137c9\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") "
Sep 15 09:32:03.640779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640593    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-tmp-dir\") pod \"a434ac87-3ccc-4156-b0e2-2b53398137c9\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") "
Sep 15 09:32:03.640779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640619    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-tmp-dir\") pod \"cdc38895-56fe-41cb-b5d8-5df27fe97798\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") "
Sep 15 09:32:03.640779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640647    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a434ac87-3ccc-4156-b0e2-2b53398137c9-tls-certs\") pod \"a434ac87-3ccc-4156-b0e2-2b53398137c9\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") "
Sep 15 09:32:03.640779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640687    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdmc9\" (UniqueName: \"kubernetes.io/projected/a434ac87-3ccc-4156-b0e2-2b53398137c9-kube-api-access-zdmc9\") pod \"a434ac87-3ccc-4156-b0e2-2b53398137c9\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") "
Sep 15 09:32:03.640779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640736    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-home\") pod \"cdc38895-56fe-41cb-b5d8-5df27fe97798\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") "
Sep 15 09:32:03.641084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640806    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc38895-56fe-41cb-b5d8-5df27fe97798-tls-certs\") pod \"cdc38895-56fe-41cb-b5d8-5df27fe97798\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") "
Sep 15 09:32:03.641084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640848    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-model-cache\") pod \"cdc38895-56fe-41cb-b5d8-5df27fe97798\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") "
Sep 15 09:32:03.641084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640896    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtvrl\" (UniqueName: \"kubernetes.io/projected/cdc38895-56fe-41cb-b5d8-5df27fe97798-kube-api-access-wtvrl\") pod \"cdc38895-56fe-41cb-b5d8-5df27fe97798\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") "
Sep 15 09:32:03.641084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640893    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-home" (OuterVolumeSpecName: "home") pod "a434ac87-3ccc-4156-b0e2-2b53398137c9" (UID: "a434ac87-3ccc-4156-b0e2-2b53398137c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.641084 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.640926    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-dshm\") pod \"a434ac87-3ccc-4156-b0e2-2b53398137c9\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") "
Sep 15 09:32:03.641343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.641121    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdc38895-56fe-41cb-b5d8-5df27fe97798" (UID: "cdc38895-56fe-41cb-b5d8-5df27fe97798"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.641343 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.641334    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-home" (OuterVolumeSpecName: "home") pod "cdc38895-56fe-41cb-b5d8-5df27fe97798" (UID: "cdc38895-56fe-41cb-b5d8-5df27fe97798"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.641701 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.641660    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a434ac87-3ccc-4156-b0e2-2b53398137c9" (UID: "a434ac87-3ccc-4156-b0e2-2b53398137c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.642411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.641847    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-dshm\") pod \"cdc38895-56fe-41cb-b5d8-5df27fe97798\" (UID: \"cdc38895-56fe-41cb-b5d8-5df27fe97798\") "
Sep 15 09:32:03.642411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.641904    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-model-cache\") pod \"a434ac87-3ccc-4156-b0e2-2b53398137c9\" (UID: \"a434ac87-3ccc-4156-b0e2-2b53398137c9\") "
Sep 15 09:32:03.642411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642231    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdc38895-56fe-41cb-b5d8-5df27fe97798" (UID: "cdc38895-56fe-41cb-b5d8-5df27fe97798"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642503    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642530    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtmw\" (UniqueName: \"kubernetes.io/projected/ea195e13-0b07-427d-9183-8c44672ed1fc-kube-api-access-hdtmw\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642546    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642618    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642633    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642646    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea195e13-0b07-427d-9183-8c44672ed1fc-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.642657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642659    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.643062 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642671    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea195e13-0b07-427d-9183-8c44672ed1fc-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.643062 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.642683    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.643274 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.643243    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a434ac87-3ccc-4156-b0e2-2b53398137c9" (UID: "a434ac87-3ccc-4156-b0e2-2b53398137c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.644379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.644317    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc38895-56fe-41cb-b5d8-5df27fe97798-kube-api-access-wtvrl" (OuterVolumeSpecName: "kube-api-access-wtvrl") pod "cdc38895-56fe-41cb-b5d8-5df27fe97798" (UID: "cdc38895-56fe-41cb-b5d8-5df27fe97798"). InnerVolumeSpecName "kube-api-access-wtvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:32:03.646464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.646350    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-dshm" (OuterVolumeSpecName: "dshm") pod "cdc38895-56fe-41cb-b5d8-5df27fe97798" (UID: "cdc38895-56fe-41cb-b5d8-5df27fe97798"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.646464 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.646422    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc38895-56fe-41cb-b5d8-5df27fe97798-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdc38895-56fe-41cb-b5d8-5df27fe97798" (UID: "cdc38895-56fe-41cb-b5d8-5df27fe97798"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:32:03.647018 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.646854    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a434ac87-3ccc-4156-b0e2-2b53398137c9-kube-api-access-zdmc9" (OuterVolumeSpecName: "kube-api-access-zdmc9") pod "a434ac87-3ccc-4156-b0e2-2b53398137c9" (UID: "a434ac87-3ccc-4156-b0e2-2b53398137c9"). InnerVolumeSpecName "kube-api-access-zdmc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:32:03.647354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.647306    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-dshm" (OuterVolumeSpecName: "dshm") pod "a434ac87-3ccc-4156-b0e2-2b53398137c9" (UID: "a434ac87-3ccc-4156-b0e2-2b53398137c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:03.647447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.647398    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a434ac87-3ccc-4156-b0e2-2b53398137c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a434ac87-3ccc-4156-b0e2-2b53398137c9" (UID: "a434ac87-3ccc-4156-b0e2-2b53398137c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744620    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc38895-56fe-41cb-b5d8-5df27fe97798-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744664    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtvrl\" (UniqueName: \"kubernetes.io/projected/cdc38895-56fe-41cb-b5d8-5df27fe97798-kube-api-access-wtvrl\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744680    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744692    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc38895-56fe-41cb-b5d8-5df27fe97798-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744705    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a434ac87-3ccc-4156-b0e2-2b53398137c9-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744717    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a434ac87-3ccc-4156-b0e2-2b53398137c9-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.744850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.744729    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmc9\" (UniqueName: \"kubernetes.io/projected/a434ac87-3ccc-4156-b0e2-2b53398137c9-kube-api-access-zdmc9\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:03.802119 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.802077    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:03.923872 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.923752    2571 generic.go:356] "Generic (PLEG): container finished" podID="ea195e13-0b07-427d-9183-8c44672ed1fc" containerID="3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec" exitCode=0
Sep 15 09:32:03.924038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.923879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv" event={"ID":"ea195e13-0b07-427d-9183-8c44672ed1fc","Type":"ContainerDied","Data":"3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec"}
Sep 15 09:32:03.924038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.923918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv" event={"ID":"ea195e13-0b07-427d-9183-8c44672ed1fc","Type":"ContainerDied","Data":"c7a70da7b3e50fc8c77f279d39f12afda4c1861fe42d77ad817eb84a8c1bcf8c"}
Sep 15 09:32:03.924038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.923943    2571 scope.go:117] "RemoveContainer" containerID="3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec"
Sep 15 09:32:03.924212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.924103    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"
Sep 15 09:32:03.932850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.931606    2571 generic.go:356] "Generic (PLEG): container finished" podID="a434ac87-3ccc-4156-b0e2-2b53398137c9" containerID="2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d" exitCode=0
Sep 15 09:32:03.932850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.931794    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"
Sep 15 09:32:03.932850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.932537    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv" event={"ID":"a434ac87-3ccc-4156-b0e2-2b53398137c9","Type":"ContainerDied","Data":"2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d"}
Sep 15 09:32:03.932850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.932579    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv" event={"ID":"a434ac87-3ccc-4156-b0e2-2b53398137c9","Type":"ContainerDied","Data":"1909bd68081a260a797f518ebc192fa29d71e55457cefd6b73523fba7c62bb9f"}
Sep 15 09:32:03.939420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.939382    2571 generic.go:356] "Generic (PLEG): container finished" podID="cdc38895-56fe-41cb-b5d8-5df27fe97798" containerID="c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734" exitCode=0
Sep 15 09:32:03.939584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.939507    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57" event={"ID":"cdc38895-56fe-41cb-b5d8-5df27fe97798","Type":"ContainerDied","Data":"c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734"}
Sep 15 09:32:03.939584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.939544    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57" event={"ID":"cdc38895-56fe-41cb-b5d8-5df27fe97798","Type":"ContainerDied","Data":"265a42aa38a8dd56e8e9bda6b5f7a7fe4a781f953c9e16d139c01ab417052b57"}
Sep 15 09:32:03.939715 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.939615    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"
Sep 15 09:32:03.946012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.945231    2571 scope.go:117] "RemoveContainer" containerID="3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec"
Sep 15 09:32:03.946012 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:32:03.945620    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec\": container with ID starting with 3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec not found: ID does not exist" containerID="3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec"
Sep 15 09:32:03.946012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.945656    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec"} err="failed to get container status \"3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec\": rpc error: code = NotFound desc = could not find container \"3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec\": container with ID starting with 3ecfe91f81ef0090a6e03353af7d235d3e2dd9bdfabecae77ba6d828d893c7ec not found: ID does not exist"
Sep 15 09:32:03.946012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.945678    2571 scope.go:117] "RemoveContainer" containerID="2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d"
Sep 15 09:32:03.967141 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.967122    2571 scope.go:117] "RemoveContainer" containerID="2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d"
Sep 15 09:32:03.967450 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:32:03.967434    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d\": container with ID starting with 2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d not found: ID does not exist" containerID="2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d"
Sep 15 09:32:03.967494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.967457    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d"} err="failed to get container status \"2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d\": rpc error: code = NotFound desc = could not find container \"2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d\": container with ID starting with 2cb9da818ef255dd3910442e136fc5d99982aa5a86a6482f91cc143ba50ad58d not found: ID does not exist"
Sep 15 09:32:03.967494 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.967476    2571 scope.go:117] "RemoveContainer" containerID="c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734"
Sep 15 09:32:03.975768 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.975742    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"]
Sep 15 09:32:03.976143 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.976125    2571 scope.go:117] "RemoveContainer" containerID="c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734"
Sep 15 09:32:03.976576 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:32:03.976551    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734\": container with ID starting with c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734 not found: ID does not exist" containerID="c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734"
Sep 15 09:32:03.976750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.976582    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734"} err="failed to get container status \"c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734\": rpc error: code = NotFound desc = could not find container \"c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734\": container with ID starting with c30d044b26737a6a5a2ecfc230b86711f1deb92cf492aa7a430fc7ffd42bc734 not found: ID does not exist"
Sep 15 09:32:03.979584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.979560    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/simulator-w0-ffb593-kserve-5f765d8494-5zs57"]
Sep 15 09:32:03.988965 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.988940    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"]
Sep 15 09:32:03.995061 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:03.995036    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/distinct-2-w0-ffb593-kserve-78c879b969-cxmbv"]
Sep 15 09:32:04.010497 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.010465    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"]
Sep 15 09:32:04.014208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.014184    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ffb593/premium-w0-ffb593-kserve-754d45d75f-qg8pv"]
Sep 15 09:32:04.055905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.055872    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a434ac87-3ccc-4156-b0e2-2b53398137c9" path="/var/lib/kubelet/pods/a434ac87-3ccc-4156-b0e2-2b53398137c9/volumes"
Sep 15 09:32:04.056328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.056315    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc38895-56fe-41cb-b5d8-5df27fe97798" path="/var/lib/kubelet/pods/cdc38895-56fe-41cb-b5d8-5df27fe97798/volumes"
Sep 15 09:32:04.056691 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.056679    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea195e13-0b07-427d-9183-8c44672ed1fc" path="/var/lib/kubelet/pods/ea195e13-0b07-427d-9183-8c44672ed1fc/volumes"
Sep 15 09:32:04.413785 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.413742    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:04.417542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.417505    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"]
Sep 15 09:32:04.422850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.421500    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ffb593-29824410-qlvsr"]
Sep 15 09:32:04.494037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.494008    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:04.554407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.554371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:04.626689 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.626631    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:04.680482 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.680370    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:04.950511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:04.950418    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:05.664807 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:05.664770    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:06.055428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:06.055346    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e15600-83ee-4412-93d6-e7f12178d6c5" path="/var/lib/kubelet/pods/02e15600-83ee-4412-93d6-e7f12178d6c5/volumes"
Sep 15 09:32:06.480231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:06.480198    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:07.268769 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:07.268735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:07.967840 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:07.967796    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:08.725616 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:08.725566    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:09.508038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:09.508002    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:10.555360 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:10.555276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:10.613639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:10.613587    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:10.823394 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:10.823312    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:12.996609 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:12.996577    2571 generic.go:356] "Generic (PLEG): container finished" podID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerID="829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074" exitCode=0
Sep 15 09:32:12.997045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:12.996652    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" event={"ID":"ddbce716-d8fe-4bb3-93eb-546af82f8b84","Type":"ContainerDied","Data":"829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"}
Sep 15 09:32:12.997121 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:12.997106    2571 scope.go:117] "RemoveContainer" containerID="829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"
Sep 15 09:32:12.998370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:12.998346    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:32:12.998443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:12.998409    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:32:12.998443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:12.998440    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:32:14.004157 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:14.004118    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" event={"ID":"ddbce716-d8fe-4bb3-93eb-546af82f8b84","Type":"ContainerStarted","Data":"ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920"}
Sep 15 09:32:14.800188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:14.800148    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:32:14.801511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:14.801481    2571 patch_prober.go:28] interesting pod/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 09:32:14.801633 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:14.801535    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 09:32:15.135724 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.135668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:15.201129 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.201093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:15.301239 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.301180    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:15.799661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.799628    2571 patch_prober.go:28] interesting pod/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 09:32:15.799867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.799682    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 09:32:15.817715 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.817681    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:15.921618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:15.921519    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:16.550335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:16.550290    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:16.800141 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:16.800104    2571 patch_prober.go:28] interesting pod/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 09:32:16.800329 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:16.800171    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 09:32:17.265684 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:17.265654    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:17.800406 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:17.800367    2571 patch_prober.go:28] interesting pod/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 09:32:17.800856 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:17.800431    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 09:32:17.929468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:17.929422    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:18.753447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:18.753414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:32:18.783853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:18.783788    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:18.799984 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:18.799949    2571 patch_prober.go:28] interesting pod/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 15 09:32:18.800158 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:18.800004    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 15 09:32:19.471556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:19.471522    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:19.803626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:19.803535    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:32:19.804729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:19.804705    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:32:20.588548 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:20.588497    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:21.304420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:21.304385    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:21.405039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:21.405010    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:21.564294 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:21.564177    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:22.002201 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:22.002128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:22.743250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:22.743223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:23.430632 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:23.430597    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:24.303181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:24.303142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:25.078186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:25.078155    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:25.443392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:25.443356    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:25.540933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:25.540903    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:25.880174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:25.880137    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:25.986449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:25.986421    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:26.798320 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:26.798277    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:27.584485 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:27.584448    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:28.336789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:28.336758    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:32.188128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:32.188098    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:37.912353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:37.912320    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:38.012626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:38.012595    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:38.103370 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:38.103335    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:38.154500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:38.154467    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:38.459104 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:38.459074    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:38.543284 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:38.543251    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:39.230195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:39.230161    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:40.080401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:40.080371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:40.880790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:40.880749    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:41.599749 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:41.599693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:42.391577 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:42.391537    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:43.804636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:43.804600    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:32:44.297280 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:44.297227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:44.396653 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:44.396613    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:44.534436 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:44.534391    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"]
Sep 15 09:32:44.534967 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:44.534937    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy" containerID="cri-o://ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920" gracePeriod=30
Sep 15 09:32:44.556401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:44.555054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:44.998231 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:44.998171    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:45.961525 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:45.961494    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:46.621489 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:46.621459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:47.686845 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:47.686786    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:48.378401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:48.378354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:49.191444 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.191408    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:49.800195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.800167    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:32:49.848769 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.848740    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p62sd\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-kube-api-access-p62sd\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.848963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.848789    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-podinfo\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.848963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.848820    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-credential-socket\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.848963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.848893    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-certs\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.848963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.848938    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-socket\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.849156 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.848993    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-envoy\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.849156 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849076    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-token\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.849156 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849106    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istiod-ca-cert\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.849156 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-data\") pod \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\" (UID: \"ddbce716-d8fe-4bb3-93eb-546af82f8b84\") "
Sep 15 09:32:49.849366 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849193    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:49.849366 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849216    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:49.849588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849523    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:49.849588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849570    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.849588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849590    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.849853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849570    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:32:49.849853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.849569    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:49.851140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.851108    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:32:49.851247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.851231    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-kube-api-access-p62sd" (OuterVolumeSpecName: "kube-api-access-p62sd") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "kube-api-access-p62sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:32:49.851725 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.851709    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:32:49.851772 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.851719    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddbce716-d8fe-4bb3-93eb-546af82f8b84" (UID: "ddbce716-d8fe-4bb3-93eb-546af82f8b84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:32:49.870484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.870459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:49.950309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950278    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.950309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950305    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.950309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950316    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.950538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950325    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.950538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950332    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.950538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950340    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62sd\" (UniqueName: \"kubernetes.io/projected/ddbce716-d8fe-4bb3-93eb-546af82f8b84-kube-api-access-p62sd\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:49.950538 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:49.950349    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbce716-d8fe-4bb3-93eb-546af82f8b84-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:32:50.233348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.233259    2571 generic.go:356] "Generic (PLEG): container finished" podID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerID="ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920" exitCode=0
Sep 15 09:32:50.233348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.233309    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"
Sep 15 09:32:50.233348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.233331    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" event={"ID":"ddbce716-d8fe-4bb3-93eb-546af82f8b84","Type":"ContainerDied","Data":"ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920"}
Sep 15 09:32:50.233810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.233372    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4" event={"ID":"ddbce716-d8fe-4bb3-93eb-546af82f8b84","Type":"ContainerDied","Data":"bc33440f9b1097a2201aef2dc43293afff34f53846b891bc409da94eae3d112b"}
Sep 15 09:32:50.233810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.233395    2571 scope.go:117] "RemoveContainer" containerID="ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920"
Sep 15 09:32:50.242755 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.242733    2571 scope.go:117] "RemoveContainer" containerID="829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"
Sep 15 09:32:50.249721 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.249693    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"]
Sep 15 09:32:50.251920 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.251903    2571 scope.go:117] "RemoveContainer" containerID="ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920"
Sep 15 09:32:50.252443 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:32:50.252377    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920\": container with ID starting with ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920 not found: ID does not exist" containerID="ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920"
Sep 15 09:32:50.252443 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.252421    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920"} err="failed to get container status \"ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920\": rpc error: code = NotFound desc = could not find container \"ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920\": container with ID starting with ae9958628866c4248c5651401e1bb86f95e57bc3695072a2489ad7124e501920 not found: ID does not exist"
Sep 15 09:32:50.252642 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.252449    2571 scope.go:117] "RemoveContainer" containerID="829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"
Sep 15 09:32:50.252799 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:32:50.252769    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074\": container with ID starting with 829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074 not found: ID does not exist" containerID="829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"
Sep 15 09:32:50.252888 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.252808    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074"} err="failed to get container status \"829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074\": rpc error: code = NotFound desc = could not find container \"829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074\": container with ID starting with 829e35d96b21a4fa7f05ccc3fbfe7a3cbd32f34b104033143feff4d0b8fb3074 not found: ID does not exist"
Sep 15 09:32:50.255203 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:50.255181    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ff390e-openshift-default-5b8f79df9b-7vnp4"]
Sep 15 09:32:52.055518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:52.055480    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" path="/var/lib/kubelet/pods/ddbce716-d8fe-4bb3-93eb-546af82f8b84/volumes"
Sep 15 09:32:55.530471 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:55.530439    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:55.628208 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:55.628176    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:55.729453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:55.729421    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:55.797264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:55.797181    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:56.090055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:56.090023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:56.190943 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:56.190915    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:56.877055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:56.877019    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:57.673412 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:57.673376    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:58.393675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:58.393641    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:32:59.182285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:32:59.182252    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:01.948048 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:01.948015    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:02.047144 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:02.047111    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:02.243483 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:02.243393    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:03.954335 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:03.954295    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:04.321120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:04.321030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:04.996365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:04.996328    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:05.492014 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:05.491972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:05.874821 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:05.874785    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:06.665011 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:06.664978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:07.502099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:07.502061    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:07.928752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:07.928719    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:08.215686 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:08.215596    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:08.907101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:08.907059    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:33:09.073250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:09.073204    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:09.795051 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:09.795014    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:10.479675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:10.479637    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"]
Sep 15 09:33:10.484896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:10.484819    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"]
Sep 15 09:33:10.496876 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:10.496814    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"]
Sep 15 09:33:10.504611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:10.504577    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"]
Sep 15 09:33:10.581508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:10.581473    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:12.688000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:12.687962    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.286917 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.286869    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"]
Sep 15 09:33:13.287147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.287121    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" podUID="f6a33b64-f722-4977-b84e-3392a87ed266" containerName="maas-api" containerID="cri-o://89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5" gracePeriod=30
Sep 15 09:33:13.311300 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.311255    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.316805 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.316774    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"]
Sep 15 09:33:13.318821 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.318801    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f032a1-29824410-tdskx"]
Sep 15 09:33:13.379858 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.377048    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.438171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.438132    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.519051 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.519021    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.539693 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.539629    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:33:13.577714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.577637    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.595880 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.595850    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdmb8\" (UniqueName: \"kubernetes.io/projected/f6a33b64-f722-4977-b84e-3392a87ed266-kube-api-access-tdmb8\") pod \"f6a33b64-f722-4977-b84e-3392a87ed266\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") "
Sep 15 09:33:13.596045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.595919    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a33b64-f722-4977-b84e-3392a87ed266-openshift-service-ca\") pod \"f6a33b64-f722-4977-b84e-3392a87ed266\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") "
Sep 15 09:33:13.596045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.595986    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-metrics-certs\") pod \"f6a33b64-f722-4977-b84e-3392a87ed266\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") "
Sep 15 09:33:13.596045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.596026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls\") pod \"f6a33b64-f722-4977-b84e-3392a87ed266\" (UID: \"f6a33b64-f722-4977-b84e-3392a87ed266\") "
Sep 15 09:33:13.596342 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.596311    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a33b64-f722-4977-b84e-3392a87ed266-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6a33b64-f722-4977-b84e-3392a87ed266" (UID: "f6a33b64-f722-4977-b84e-3392a87ed266"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:33:13.596466 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.596429    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a33b64-f722-4977-b84e-3392a87ed266-openshift-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:13.598182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.598151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a33b64-f722-4977-b84e-3392a87ed266-kube-api-access-tdmb8" (OuterVolumeSpecName: "kube-api-access-tdmb8") pod "f6a33b64-f722-4977-b84e-3392a87ed266" (UID: "f6a33b64-f722-4977-b84e-3392a87ed266"). InnerVolumeSpecName "kube-api-access-tdmb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:13.598272 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.598179    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6a33b64-f722-4977-b84e-3392a87ed266" (UID: "f6a33b64-f722-4977-b84e-3392a87ed266"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:13.598272 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.598201    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6a33b64-f722-4977-b84e-3392a87ed266" (UID: "f6a33b64-f722-4977-b84e-3392a87ed266"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:13.697560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.697523    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-metrics-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:13.697560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.697556    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a33b64-f722-4977-b84e-3392a87ed266-maas-api-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:13.697560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.697567    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmb8\" (UniqueName: \"kubernetes.io/projected/f6a33b64-f722-4977-b84e-3392a87ed266-kube-api-access-tdmb8\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:13.846596 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.846542    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:13.990712 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:13.990671    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:14.055424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.055388    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f262fc-babb-4f5e-8e94-5156f738cc3e" path="/var/lib/kubelet/pods/94f262fc-babb-4f5e-8e94-5156f738cc3e/volumes"
Sep 15 09:33:14.372989 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.372957    2571 generic.go:356] "Generic (PLEG): container finished" podID="f6a33b64-f722-4977-b84e-3392a87ed266" containerID="89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5" exitCode=0
Sep 15 09:33:14.373197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.373004    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"
Sep 15 09:33:14.373197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.373041    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" event={"ID":"f6a33b64-f722-4977-b84e-3392a87ed266","Type":"ContainerDied","Data":"89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5"}
Sep 15 09:33:14.373197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.373100    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f" event={"ID":"f6a33b64-f722-4977-b84e-3392a87ed266","Type":"ContainerDied","Data":"642e6675b6ca4d09d08548a1d1d53d2e8c4682dc430a2c048265a9ed19904563"}
Sep 15 09:33:14.373197 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.373124    2571 scope.go:117] "RemoveContainer" containerID="89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5"
Sep 15 09:33:14.385518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.385498    2571 scope.go:117] "RemoveContainer" containerID="89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5"
Sep 15 09:33:14.385855 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:14.385821    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5\": container with ID starting with 89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5 not found: ID does not exist" containerID="89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5"
Sep 15 09:33:14.385929 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.385868    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5"} err="failed to get container status \"89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5\": rpc error: code = NotFound desc = could not find container \"89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5\": container with ID starting with 89851c1bf3277fb34fcde0b6a275762a6bc0febd597fd152df12c56af84daac5 not found: ID does not exist"
Sep 15 09:33:14.393654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.393625    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"]
Sep 15 09:33:14.397550 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.397524    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f032a1-6b9c56687b-9c26f"]
Sep 15 09:33:14.944982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:14.944947    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:15.653363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:15.653322    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:16.055996 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:16.055915    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a33b64-f722-4977-b84e-3392a87ed266" path="/var/lib/kubelet/pods/f6a33b64-f722-4977-b84e-3392a87ed266/volumes"
Sep 15 09:33:16.289492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:16.289444    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:19.658547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:19.658498    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:19.759263 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:19.759230    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:19.907630 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:19.907590    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"]
Sep 15 09:33:19.907905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:19.907854    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy" containerID="cri-o://abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146" gracePeriod=30
Sep 15 09:33:19.934279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:19.934180    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:21.920250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:21.920156    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:21.920250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:21.920245    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"]
Sep 15 09:33:21.922306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:21.922277    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-890f3e-29824410-bs58t"]
Sep 15 09:33:22.010334 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:22.010292    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:22.055317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:22.055283    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b397415-b7d9-4d02-ad7a-a4ac043988cc" path="/var/lib/kubelet/pods/0b397415-b7d9-4d02-ad7a-a4ac043988cc/volumes"
Sep 15 09:33:22.111207 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:22.111175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:22.174367 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:22.174252    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:22.443217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:22.443112    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:23.173357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:23.173297    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:23.892428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:23.892394    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:25.166518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.166491    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:33:25.220810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220705    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56abe022-36c3-4004-91a8-ed166248ea47-istio-podinfo\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.220810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220757    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-data\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.220810 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220795    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-socket\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220855    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prn6p\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-kube-api-access-prn6p\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220887    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-istio-token\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220944    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-envoy\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.220983    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-credential-socket\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221029    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56abe022-36c3-4004-91a8-ed166248ea47-istiod-ca-cert\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221122    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-certs\") pod \"56abe022-36c3-4004-91a8-ed166248ea47\" (UID: \"56abe022-36c3-4004-91a8-ed166248ea47\") "
Sep 15 09:33:25.221483 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221122    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-data" (OuterVolumeSpecName: "istio-data") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.221483 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221137    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.221483 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.221639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221481    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56abe022-36c3-4004-91a8-ed166248ea47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:33:25.221639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221483    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.221842 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221806    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.221923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221845    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.221923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221858    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.221923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221872    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56abe022-36c3-4004-91a8-ed166248ea47-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.221923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.221885    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.223304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.223272    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.223619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.223591    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-kube-api-access-prn6p" (OuterVolumeSpecName: "kube-api-access-prn6p") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "kube-api-access-prn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:25.223719 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.223624    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-istio-token" (OuterVolumeSpecName: "istio-token") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:25.223719 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.223672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56abe022-36c3-4004-91a8-ed166248ea47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56abe022-36c3-4004-91a8-ed166248ea47" (UID: "56abe022-36c3-4004-91a8-ed166248ea47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:33:25.322963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.322925    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56abe022-36c3-4004-91a8-ed166248ea47-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.322963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.322955    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56abe022-36c3-4004-91a8-ed166248ea47-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.322963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.322967    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prn6p\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-kube-api-access-prn6p\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.322963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.322976    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56abe022-36c3-4004-91a8-ed166248ea47-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.437106 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.437072    2571 generic.go:356] "Generic (PLEG): container finished" podID="56abe022-36c3-4004-91a8-ed166248ea47" containerID="abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146" exitCode=0
Sep 15 09:33:25.437281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.437141    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"
Sep 15 09:33:25.437281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.437158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" event={"ID":"56abe022-36c3-4004-91a8-ed166248ea47","Type":"ContainerDied","Data":"abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146"}
Sep 15 09:33:25.437281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.437237    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd" event={"ID":"56abe022-36c3-4004-91a8-ed166248ea47","Type":"ContainerDied","Data":"a75fbbda8189e71c5199860a5c93651f70322746c653f13aa5e7ec399373a752"}
Sep 15 09:33:25.437281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.437260    2571 scope.go:117] "RemoveContainer" containerID="abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146"
Sep 15 09:33:25.446871 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.446853    2571 scope.go:117] "RemoveContainer" containerID="25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"
Sep 15 09:33:25.456428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.456395    2571 scope.go:117] "RemoveContainer" containerID="abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146"
Sep 15 09:33:25.456991 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:25.456962    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146\": container with ID starting with abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146 not found: ID does not exist" containerID="abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146"
Sep 15 09:33:25.457098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.457000    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146"} err="failed to get container status \"abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146\": rpc error: code = NotFound desc = could not find container \"abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146\": container with ID starting with abded7df732576f8e995d60a3547b7d595a144a6991106b98a223762e676b146 not found: ID does not exist"
Sep 15 09:33:25.457098 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.457026    2571 scope.go:117] "RemoveContainer" containerID="25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"
Sep 15 09:33:25.457348 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:25.457328    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812\": container with ID starting with 25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812 not found: ID does not exist" containerID="25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"
Sep 15 09:33:25.457392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.457354    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812"} err="failed to get container status \"25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812\": rpc error: code = NotFound desc = could not find container \"25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812\": container with ID starting with 25d21b64cb775a0dc89c6c263c3af620b200f1ee6cbd555921e7eb57b1c0c812 not found: ID does not exist"
Sep 15 09:33:25.459166 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.459146    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"]
Sep 15 09:33:25.462970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.462949    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f032a1-openshift-default-5f4bb9c4bb-xl5hd"]
Sep 15 09:33:25.500400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.500295    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf" podUID="143d6821-6aff-4ab0-9e07-0b52eec93279" containerName="main" containerID="cri-o://c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153" gracePeriod=45
Sep 15 09:33:25.504950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.504913    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b" podUID="dd9b79dd-3444-4992-b16a-c68f9c608687" containerName="main" containerID="cri-o://2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3" gracePeriod=45
Sep 15 09:33:25.516633 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.516599    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" podUID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" containerName="main" containerID="cri-o://b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64" gracePeriod=45
Sep 15 09:33:25.524759 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.524725    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf" podUID="3e2bb914-6216-4da1-aab8-6ae238e7116a" containerName="main" containerID="cri-o://96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095" gracePeriod=45
Sep 15 09:33:25.783409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.783377    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:33:25.827031 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.826992    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-home\") pod \"dd9b79dd-3444-4992-b16a-c68f9c608687\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") "
Sep 15 09:33:25.827193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.827084    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-model-cache\") pod \"dd9b79dd-3444-4992-b16a-c68f9c608687\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") "
Sep 15 09:33:25.827193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.827161    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-dshm\") pod \"dd9b79dd-3444-4992-b16a-c68f9c608687\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") "
Sep 15 09:33:25.827318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.827195    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b79dd-3444-4992-b16a-c68f9c608687-tls-certs\") pod \"dd9b79dd-3444-4992-b16a-c68f9c608687\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") "
Sep 15 09:33:25.827318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.827256    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx8xp\" (UniqueName: \"kubernetes.io/projected/dd9b79dd-3444-4992-b16a-c68f9c608687-kube-api-access-zx8xp\") pod \"dd9b79dd-3444-4992-b16a-c68f9c608687\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") "
Sep 15 09:33:25.827420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.827324    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-tmp-dir\") pod \"dd9b79dd-3444-4992-b16a-c68f9c608687\" (UID: \"dd9b79dd-3444-4992-b16a-c68f9c608687\") "
Sep 15 09:33:25.827781 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.827753    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-home" (OuterVolumeSpecName: "home") pod "dd9b79dd-3444-4992-b16a-c68f9c608687" (UID: "dd9b79dd-3444-4992-b16a-c68f9c608687"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.828539 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.828512    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd9b79dd-3444-4992-b16a-c68f9c608687" (UID: "dd9b79dd-3444-4992-b16a-c68f9c608687"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.828648 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.828527    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd9b79dd-3444-4992-b16a-c68f9c608687" (UID: "dd9b79dd-3444-4992-b16a-c68f9c608687"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.833242 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.833213    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-dshm" (OuterVolumeSpecName: "dshm") pod "dd9b79dd-3444-4992-b16a-c68f9c608687" (UID: "dd9b79dd-3444-4992-b16a-c68f9c608687"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.833385 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.833256    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9b79dd-3444-4992-b16a-c68f9c608687-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd9b79dd-3444-4992-b16a-c68f9c608687" (UID: "dd9b79dd-3444-4992-b16a-c68f9c608687"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:25.835806 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.835764    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9b79dd-3444-4992-b16a-c68f9c608687-kube-api-access-zx8xp" (OuterVolumeSpecName: "kube-api-access-zx8xp") pod "dd9b79dd-3444-4992-b16a-c68f9c608687" (UID: "dd9b79dd-3444-4992-b16a-c68f9c608687"). InnerVolumeSpecName "kube-api-access-zx8xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:25.838337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.838315    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:33:25.845175 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.845154    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:33:25.848870 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.848816    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928723    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-home\") pod \"143d6821-6aff-4ab0-9e07-0b52eec93279\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928774    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-tmp-dir\") pod \"143d6821-6aff-4ab0-9e07-0b52eec93279\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928804    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-home\") pod \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-model-cache\") pod \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928914    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-model-cache\") pod \"3e2bb914-6216-4da1-aab8-6ae238e7116a\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928945    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk6pj\" (UniqueName: \"kubernetes.io/projected/143d6821-6aff-4ab0-9e07-0b52eec93279-kube-api-access-pk6pj\") pod \"143d6821-6aff-4ab0-9e07-0b52eec93279\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928975    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-home\") pod \"3e2bb914-6216-4da1-aab8-6ae238e7116a\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.928996    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-model-cache\") pod \"143d6821-6aff-4ab0-9e07-0b52eec93279\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929023    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-tmp-dir\") pod \"3e2bb914-6216-4da1-aab8-6ae238e7116a\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929054    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg8br\" (UniqueName: \"kubernetes.io/projected/3e2bb914-6216-4da1-aab8-6ae238e7116a-kube-api-access-jg8br\") pod \"3e2bb914-6216-4da1-aab8-6ae238e7116a\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-home" (OuterVolumeSpecName: "home") pod "143d6821-6aff-4ab0-9e07-0b52eec93279" (UID: "143d6821-6aff-4ab0-9e07-0b52eec93279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929081    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-dshm\") pod \"3e2bb914-6216-4da1-aab8-6ae238e7116a\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929113    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-home" (OuterVolumeSpecName: "home") pod "5a3d2dd1-03af-4c21-b9c0-d511cf199807" (UID: "5a3d2dd1-03af-4c21-b9c0-d511cf199807"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929133    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tmp-dir\") pod \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929180    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tls-certs\") pod \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") "
Sep 15 09:33:25.929217 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929208    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-dshm\") pod \"143d6821-6aff-4ab0-9e07-0b52eec93279\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") "
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929236    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-model-cache" (OuterVolumeSpecName: "model-cache") pod "143d6821-6aff-4ab0-9e07-0b52eec93279" (UID: "143d6821-6aff-4ab0-9e07-0b52eec93279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929252    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2bb914-6216-4da1-aab8-6ae238e7116a-tls-certs\") pod \"3e2bb914-6216-4da1-aab8-6ae238e7116a\" (UID: \"3e2bb914-6216-4da1-aab8-6ae238e7116a\") "
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929290    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-dshm\") pod \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") "
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929326    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh6xz\" (UniqueName: \"kubernetes.io/projected/5a3d2dd1-03af-4c21-b9c0-d511cf199807-kube-api-access-qh6xz\") pod \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\" (UID: \"5a3d2dd1-03af-4c21-b9c0-d511cf199807\") "
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929342    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e2bb914-6216-4da1-aab8-6ae238e7116a" (UID: "3e2bb914-6216-4da1-aab8-6ae238e7116a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929358    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143d6821-6aff-4ab0-9e07-0b52eec93279-tls-certs\") pod \"143d6821-6aff-4ab0-9e07-0b52eec93279\" (UID: \"143d6821-6aff-4ab0-9e07-0b52eec93279\") "
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929431    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a3d2dd1-03af-4c21-b9c0-d511cf199807" (UID: "5a3d2dd1-03af-4c21-b9c0-d511cf199807"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929604    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e2bb914-6216-4da1-aab8-6ae238e7116a" (UID: "3e2bb914-6216-4da1-aab8-6ae238e7116a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.930174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.929630    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-home" (OuterVolumeSpecName: "home") pod "3e2bb914-6216-4da1-aab8-6ae238e7116a" (UID: "3e2bb914-6216-4da1-aab8-6ae238e7116a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930212    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930252    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930270    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930283    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930296    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930309    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930343    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930357    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930370    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930385    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b79dd-3444-4992-b16a-c68f9c608687-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930417    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8xp\" (UniqueName: \"kubernetes.io/projected/dd9b79dd-3444-4992-b16a-c68f9c608687-kube-api-access-zx8xp\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930433    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b79dd-3444-4992-b16a-c68f9c608687-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.930705 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930447    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:25.932401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.930739    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "143d6821-6aff-4ab0-9e07-0b52eec93279" (UID: "143d6821-6aff-4ab0-9e07-0b52eec93279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.932401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.931032    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a3d2dd1-03af-4c21-b9c0-d511cf199807" (UID: "5a3d2dd1-03af-4c21-b9c0-d511cf199807"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.932401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.932204    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2bb914-6216-4da1-aab8-6ae238e7116a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e2bb914-6216-4da1-aab8-6ae238e7116a" (UID: "3e2bb914-6216-4da1-aab8-6ae238e7116a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:25.934508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.932527    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143d6821-6aff-4ab0-9e07-0b52eec93279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "143d6821-6aff-4ab0-9e07-0b52eec93279" (UID: "143d6821-6aff-4ab0-9e07-0b52eec93279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:25.934508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.932706    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-dshm" (OuterVolumeSpecName: "dshm") pod "3e2bb914-6216-4da1-aab8-6ae238e7116a" (UID: "3e2bb914-6216-4da1-aab8-6ae238e7116a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.934508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.934254    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a3d2dd1-03af-4c21-b9c0-d511cf199807" (UID: "5a3d2dd1-03af-4c21-b9c0-d511cf199807"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:33:25.934508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.934464    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-dshm" (OuterVolumeSpecName: "dshm") pod "5a3d2dd1-03af-4c21-b9c0-d511cf199807" (UID: "5a3d2dd1-03af-4c21-b9c0-d511cf199807"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.935308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.935199    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3d2dd1-03af-4c21-b9c0-d511cf199807-kube-api-access-qh6xz" (OuterVolumeSpecName: "kube-api-access-qh6xz") pod "5a3d2dd1-03af-4c21-b9c0-d511cf199807" (UID: "5a3d2dd1-03af-4c21-b9c0-d511cf199807"). InnerVolumeSpecName "kube-api-access-qh6xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:25.935308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.935180    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-dshm" (OuterVolumeSpecName: "dshm") pod "143d6821-6aff-4ab0-9e07-0b52eec93279" (UID: "143d6821-6aff-4ab0-9e07-0b52eec93279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:25.935484 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.935343    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143d6821-6aff-4ab0-9e07-0b52eec93279-kube-api-access-pk6pj" (OuterVolumeSpecName: "kube-api-access-pk6pj") pod "143d6821-6aff-4ab0-9e07-0b52eec93279" (UID: "143d6821-6aff-4ab0-9e07-0b52eec93279"). InnerVolumeSpecName "kube-api-access-pk6pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:25.935647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:25.935559    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2bb914-6216-4da1-aab8-6ae238e7116a-kube-api-access-jg8br" (OuterVolumeSpecName: "kube-api-access-jg8br") pod "3e2bb914-6216-4da1-aab8-6ae238e7116a" (UID: "3e2bb914-6216-4da1-aab8-6ae238e7116a"). InnerVolumeSpecName "kube-api-access-jg8br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031422    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031457    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6xz\" (UniqueName: \"kubernetes.io/projected/5a3d2dd1-03af-4c21-b9c0-d511cf199807-kube-api-access-qh6xz\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031468    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143d6821-6aff-4ab0-9e07-0b52eec93279-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031477    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031486    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a3d2dd1-03af-4c21-b9c0-d511cf199807-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031493    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6pj\" (UniqueName: \"kubernetes.io/projected/143d6821-6aff-4ab0-9e07-0b52eec93279-kube-api-access-pk6pj\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031501    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8br\" (UniqueName: \"kubernetes.io/projected/3e2bb914-6216-4da1-aab8-6ae238e7116a-kube-api-access-jg8br\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031509    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2bb914-6216-4da1-aab8-6ae238e7116a-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031897 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031518    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a3d2dd1-03af-4c21-b9c0-d511cf199807-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031897 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031525    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143d6821-6aff-4ab0-9e07-0b52eec93279-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.031897 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.031532    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2bb914-6216-4da1-aab8-6ae238e7116a-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:26.056014 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.055983    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56abe022-36c3-4004-91a8-ed166248ea47" path="/var/lib/kubelet/pods/56abe022-36c3-4004-91a8-ed166248ea47/volumes"
Sep 15 09:33:26.444191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.444153    2571 generic.go:356] "Generic (PLEG): container finished" podID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" containerID="b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64" exitCode=0
Sep 15 09:33:26.444744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.444198    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"
Sep 15 09:33:26.444744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.444243    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" event={"ID":"5a3d2dd1-03af-4c21-b9c0-d511cf199807","Type":"ContainerDied","Data":"b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64"}
Sep 15 09:33:26.444744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.444291    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" event={"ID":"5a3d2dd1-03af-4c21-b9c0-d511cf199807","Type":"ContainerDied","Data":"46ff71816b45bbef565dda66e3250c95d1509cbf1746fd2db82cc9d89fa2bb8b"}
Sep 15 09:33:26.444744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.444316    2571 scope.go:117] "RemoveContainer" containerID="b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64"
Sep 15 09:33:26.447230 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.447200    2571 generic.go:356] "Generic (PLEG): container finished" podID="143d6821-6aff-4ab0-9e07-0b52eec93279" containerID="c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153" exitCode=0
Sep 15 09:33:26.447355 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.447234    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"
Sep 15 09:33:26.447355 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.447320    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf" event={"ID":"143d6821-6aff-4ab0-9e07-0b52eec93279","Type":"ContainerDied","Data":"c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153"}
Sep 15 09:33:26.447355 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.447353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf" event={"ID":"143d6821-6aff-4ab0-9e07-0b52eec93279","Type":"ContainerDied","Data":"6297a6a3282a69ab06efa53e051260bcc2f955f7e5e79a229e544ec7332266ee"}
Sep 15 09:33:26.448651 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.448632    2571 generic.go:356] "Generic (PLEG): container finished" podID="3e2bb914-6216-4da1-aab8-6ae238e7116a" containerID="96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095" exitCode=0
Sep 15 09:33:26.448744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.448665    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"
Sep 15 09:33:26.448744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.448710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf" event={"ID":"3e2bb914-6216-4da1-aab8-6ae238e7116a","Type":"ContainerDied","Data":"96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095"}
Sep 15 09:33:26.448892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.448753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf" event={"ID":"3e2bb914-6216-4da1-aab8-6ae238e7116a","Type":"ContainerDied","Data":"d7ffe601b652524ca09d68c4327f4cf5617ee6e700571435fbae338e8c4b0728"}
Sep 15 09:33:26.450154 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.450126    2571 generic.go:356] "Generic (PLEG): container finished" podID="dd9b79dd-3444-4992-b16a-c68f9c608687" containerID="2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3" exitCode=0
Sep 15 09:33:26.450154 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.450151    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"
Sep 15 09:33:26.450279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.450168    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b" event={"ID":"dd9b79dd-3444-4992-b16a-c68f9c608687","Type":"ContainerDied","Data":"2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3"}
Sep 15 09:33:26.450279 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.450194    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b" event={"ID":"dd9b79dd-3444-4992-b16a-c68f9c608687","Type":"ContainerDied","Data":"31b9898858c6476b9cf95deea5fe030b710b6e558ed77c3fb59351f0a0bc258a"}
Sep 15 09:33:26.454897 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.454881    2571 scope.go:117] "RemoveContainer" containerID="b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64"
Sep 15 09:33:26.455129 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:26.455114    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64\": container with ID starting with b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64 not found: ID does not exist" containerID="b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64"
Sep 15 09:33:26.455185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.455135    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64"} err="failed to get container status \"b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64\": rpc error: code = NotFound desc = could not find container \"b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64\": container with ID starting with b675e577257129167410543bfceb759f5cabadb48b48e1468e91251377a00b64 not found: ID does not exist"
Sep 15 09:33:26.455185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.455156    2571 scope.go:117] "RemoveContainer" containerID="c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153"
Sep 15 09:33:26.460938 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.460917    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"]
Sep 15 09:33:26.465052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.465031    2571 scope.go:117] "RemoveContainer" containerID="c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153"
Sep 15 09:33:26.465140 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.465122    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd"]
Sep 15 09:33:26.465366 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:26.465347    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153\": container with ID starting with c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153 not found: ID does not exist" containerID="c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153"
Sep 15 09:33:26.465422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.465372    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153"} err="failed to get container status \"c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153\": rpc error: code = NotFound desc = could not find container \"c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153\": container with ID starting with c074585cdfa95886ad25178db2e99c07d54b5de3764ca4adf7f1739c49bbd153 not found: ID does not exist"
Sep 15 09:33:26.465422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.465388    2571 scope.go:117] "RemoveContainer" containerID="96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095"
Sep 15 09:33:26.475904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.475881    2571 scope.go:117] "RemoveContainer" containerID="96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095"
Sep 15 09:33:26.476244 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:26.476221    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095\": container with ID starting with 96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095 not found: ID does not exist" containerID="96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095"
Sep 15 09:33:26.476306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.476254    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095"} err="failed to get container status \"96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095\": rpc error: code = NotFound desc = could not find container \"96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095\": container with ID starting with 96f6543d72a1483d084c489afdfeb70c5b681fadca31011068303b5071f62095 not found: ID does not exist"
Sep 15 09:33:26.476306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.476277    2571 scope.go:117] "RemoveContainer" containerID="2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3"
Sep 15 09:33:26.476890 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.476850    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"]
Sep 15 09:33:26.479338 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.479315    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/simulator-w4-890f3e-kserve-5d58dd5d7d-xbwwf"]
Sep 15 09:33:26.484583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.484567    2571 scope.go:117] "RemoveContainer" containerID="2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3"
Sep 15 09:33:26.484876 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:26.484855    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3\": container with ID starting with 2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3 not found: ID does not exist" containerID="2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3"
Sep 15 09:33:26.484926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.484884    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3"} err="failed to get container status \"2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3\": rpc error: code = NotFound desc = could not find container \"2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3\": container with ID starting with 2072be94633627d77756ded39e56970a14623c034ec7553df6409be450c4dbf3 not found: ID does not exist"
Sep 15 09:33:26.489549 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.489524    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"]
Sep 15 09:33:26.492930 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.492910    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-w4-890f3e-kserve-785755c694-ktc4b"]
Sep 15 09:33:26.502762 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.502737    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"]
Sep 15 09:33:26.506526 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:26.506502    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-890f3e/distinct-2-w4-890f3e-kserve-74c6c587b5-wmhsf"]
Sep 15 09:33:28.055353 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.055319    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="143d6821-6aff-4ab0-9e07-0b52eec93279" path="/var/lib/kubelet/pods/143d6821-6aff-4ab0-9e07-0b52eec93279/volumes"
Sep 15 09:33:28.055725 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.055711    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2bb914-6216-4da1-aab8-6ae238e7116a" path="/var/lib/kubelet/pods/3e2bb914-6216-4da1-aab8-6ae238e7116a/volumes"
Sep 15 09:33:28.056089 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.056076    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" path="/var/lib/kubelet/pods/5a3d2dd1-03af-4c21-b9c0-d511cf199807/volumes"
Sep 15 09:33:28.056720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.056705    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd9b79dd-3444-4992-b16a-c68f9c608687" path="/var/lib/kubelet/pods/dd9b79dd-3444-4992-b16a-c68f9c608687/volumes"
Sep 15 09:33:28.279698 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.279660    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:28.379904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.379873    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:28.516761 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.516725    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"]
Sep 15 09:33:28.517091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.517056    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy" containerID="cri-o://2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6" gracePeriod=30
Sep 15 09:33:28.541246 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:28.541210    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:30.826483 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:30.826429    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-890f3e/premium-w4-890f3e-kserve-56b4c566c9-4zlxd" podUID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Sep 15 09:33:33.766906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.766877    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:33:33.912175 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912139    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25538817-dc63-42ee-a527-35d8b90c0207-istiod-ca-cert\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912175 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912181    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-envoy\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912221    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-certs\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912247    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25538817-dc63-42ee-a527-35d8b90c0207-istio-podinfo\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912266    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-socket\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912286    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-data\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912327    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-credential-socket\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912392 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912354    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-istio-token\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912395    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rtvc\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-kube-api-access-6rtvc\") pod \"25538817-dc63-42ee-a527-35d8b90c0207\" (UID: \"25538817-dc63-42ee-a527-35d8b90c0207\") "
Sep 15 09:33:33.912710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912564    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:33.912710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912668    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-data" (OuterVolumeSpecName: "istio-data") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:33.912710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912700    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25538817-dc63-42ee-a527-35d8b90c0207-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:33:33.912959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912798    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25538817-dc63-42ee-a527-35d8b90c0207-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:33.912959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912800    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:33.912959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912811    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:33.912959 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912877    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:33.913118 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.912981    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:33.914735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.914706    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-istio-token" (OuterVolumeSpecName: "istio-token") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:33.914735 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.914711    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-kube-api-access-6rtvc" (OuterVolumeSpecName: "kube-api-access-6rtvc") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "kube-api-access-6rtvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:33.915027 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.915006    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25538817-dc63-42ee-a527-35d8b90c0207-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:33:33.915095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:33.915017    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25538817-dc63-42ee-a527-35d8b90c0207" (UID: "25538817-dc63-42ee-a527-35d8b90c0207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:33:34.013573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.013526    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:34.013573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.013566    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25538817-dc63-42ee-a527-35d8b90c0207-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:34.013573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.013575    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:34.013573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.013583    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25538817-dc63-42ee-a527-35d8b90c0207-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:34.013927 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.013593    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:34.013927 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.013601    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtvc\" (UniqueName: \"kubernetes.io/projected/25538817-dc63-42ee-a527-35d8b90c0207-kube-api-access-6rtvc\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:34.106265 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.106232    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:33:34.499024 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.498985    2571 generic.go:356] "Generic (PLEG): container finished" podID="25538817-dc63-42ee-a527-35d8b90c0207" containerID="2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6" exitCode=0
Sep 15 09:33:34.499177 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.499050    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"
Sep 15 09:33:34.499177 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.499072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" event={"ID":"25538817-dc63-42ee-a527-35d8b90c0207","Type":"ContainerDied","Data":"2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6"}
Sep 15 09:33:34.499177 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.499129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk" event={"ID":"25538817-dc63-42ee-a527-35d8b90c0207","Type":"ContainerDied","Data":"6992d5b9fe3087065b48759055a58bad7c51c20e1e9d6b361c0244020cf957f1"}
Sep 15 09:33:34.499177 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.499151    2571 scope.go:117] "RemoveContainer" containerID="2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6"
Sep 15 09:33:34.513135 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.513111    2571 scope.go:117] "RemoveContainer" containerID="ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"
Sep 15 09:33:34.518745 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.518714    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"]
Sep 15 09:33:34.521457 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.521434    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-890f3e-openshift-default-6844dc9c86-vsbqk"]
Sep 15 09:33:34.522912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.522892    2571 scope.go:117] "RemoveContainer" containerID="2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6"
Sep 15 09:33:34.523216 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:34.523194    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6\": container with ID starting with 2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6 not found: ID does not exist" containerID="2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6"
Sep 15 09:33:34.523267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.523225    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6"} err="failed to get container status \"2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6\": rpc error: code = NotFound desc = could not find container \"2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6\": container with ID starting with 2275c07e3bace077f70d9034aafbdc0ed4b7f4899b83c6e1ede0cee01ae88cf6 not found: ID does not exist"
Sep 15 09:33:34.523267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.523243    2571 scope.go:117] "RemoveContainer" containerID="ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"
Sep 15 09:33:34.523528 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:34.523509    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4\": container with ID starting with ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4 not found: ID does not exist" containerID="ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"
Sep 15 09:33:34.523563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:34.523536    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4"} err="failed to get container status \"ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4\": rpc error: code = NotFound desc = could not find container \"ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4\": container with ID starting with ff861a8aec93e209433e66f9bddbee1304e255888be525f3f4dae424bb3367c4 not found: ID does not exist"
Sep 15 09:33:35.817366 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:35.817326    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:36.055818 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:36.055780    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25538817-dc63-42ee-a527-35d8b90c0207" path="/var/lib/kubelet/pods/25538817-dc63-42ee-a527-35d8b90c0207/volumes"
Sep 15 09:33:47.209567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:47.209481    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:33:48.008314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:48.008277    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:48.108357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:48.108327    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:48.419096 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:48.419058    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:33:50.254839 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.254786    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-14b71f"]
Sep 15 09:33:50.255222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255192    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy"
Sep 15 09:33:50.255222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255206    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy"
Sep 15 09:33:50.255222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255214    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="143d6821-6aff-4ab0-9e07-0b52eec93279" containerName="main"
Sep 15 09:33:50.255222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255220    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="143d6821-6aff-4ab0-9e07-0b52eec93279" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255234    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255240    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255246    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255253    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255264    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a434ac87-3ccc-4156-b0e2-2b53398137c9" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255273    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a434ac87-3ccc-4156-b0e2-2b53398137c9" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255278    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a33b64-f722-4977-b84e-3392a87ed266" containerName="maas-api"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255283    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a33b64-f722-4977-b84e-3392a87ed266" containerName="maas-api"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255290    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255295    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255303    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9b79dd-3444-4992-b16a-c68f9c608687" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255308    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9b79dd-3444-4992-b16a-c68f9c608687" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255318    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255323    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255328    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2bb914-6216-4da1-aab8-6ae238e7116a" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255336    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2bb914-6216-4da1-aab8-6ae238e7116a" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255346    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea195e13-0b07-427d-9183-8c44672ed1fc" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255352    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea195e13-0b07-427d-9183-8c44672ed1fc" containerName="main"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255359    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255364    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy"
Sep 15 09:33:50.255361 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255370    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc38895-56fe-41cb-b5d8-5df27fe97798" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255375    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc38895-56fe-41cb-b5d8-5df27fe97798" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255437    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc38895-56fe-41cb-b5d8-5df27fe97798" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255447    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a434ac87-3ccc-4156-b0e2-2b53398137c9" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255454    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255459    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="25538817-dc63-42ee-a527-35d8b90c0207" containerName="istio-proxy"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255464    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="143d6821-6aff-4ab0-9e07-0b52eec93279" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255469    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9b79dd-3444-4992-b16a-c68f9c608687" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255474    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2bb914-6216-4da1-aab8-6ae238e7116a" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255478    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255483    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbce716-d8fe-4bb3-93eb-546af82f8b84" containerName="istio-proxy"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255492    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3d2dd1-03af-4c21-b9c0-d511cf199807" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255500    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255507    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea195e13-0b07-427d-9183-8c44672ed1fc" containerName="main"
Sep 15 09:33:50.256015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.255511    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a33b64-f722-4977-b84e-3392a87ed266" containerName="maas-api"
Sep 15 09:33:50.258373 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.258356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:50.261713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.261688    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7g2n6\""
Sep 15 09:33:50.263230 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.263206    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-14b71f"]
Sep 15 09:33:50.271766 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.271740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtzm\" (UniqueName: \"kubernetes.io/projected/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81-kube-api-access-2dtzm\") pod \"test-curl-7741-14b71f\" (UID: \"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81\") " pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:50.372791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.372755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtzm\" (UniqueName: \"kubernetes.io/projected/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81-kube-api-access-2dtzm\") pod \"test-curl-7741-14b71f\" (UID: \"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81\") " pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:50.380597 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.380568    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtzm\" (UniqueName: \"kubernetes.io/projected/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81-kube-api-access-2dtzm\") pod \"test-curl-7741-14b71f\" (UID: \"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81\") " pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:50.570483 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.570379    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:50.842072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:50.842043    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-14b71f"]
Sep 15 09:33:50.844164 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:33:50.844138    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc551cca_8f1e_4871_8f4b_3bf6a4be2b81.slice/crio-ab81b15eb37a18014698eaabb6a8318dbc30756441ae92c3350443e76327cbdf WatchSource:0}: Error finding container ab81b15eb37a18014698eaabb6a8318dbc30756441ae92c3350443e76327cbdf: Status 404 returned error can't find the container with id ab81b15eb37a18014698eaabb6a8318dbc30756441ae92c3350443e76327cbdf
Sep 15 09:33:51.593087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:51.593048    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-14b71f" event={"ID":"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81","Type":"ContainerStarted","Data":"8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b"}
Sep 15 09:33:51.593087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:51.593093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-14b71f" event={"ID":"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81","Type":"ContainerStarted","Data":"ab81b15eb37a18014698eaabb6a8318dbc30756441ae92c3350443e76327cbdf"}
Sep 15 09:33:51.607947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:51.607900    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7741-14b71f" podStartSLOduration=1.607886257 podStartE2EDuration="1.607886257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:33:51.607502441 +0000 UTC m=+1624.066791982" watchObservedRunningTime="2026-09-15 09:33:51.607886257 +0000 UTC m=+1624.067175797"
Sep 15 09:33:52.196933 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:52.196892    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-14b71f"]
Sep 15 09:33:52.201013 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:52.200984    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-14b71f"]
Sep 15 09:33:52.599354 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:52.599321    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7741-14b71f" container="test-curl-7741-14b71f"
Sep 15 09:33:53.603649 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:53.603610    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7741-14b71f" podUID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" containerName="test-curl-7741-14b71f" containerID="cri-o://8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b" gracePeriod=2
Sep 15 09:33:53.605975 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:53.605947    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-14b71f\" is forbidden: User \"system:node:ip-10-0-134-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-28.ec2.internal' and this object" podUID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:55.776228 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:55.776200    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:55.825290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:55.825252    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dtzm\" (UniqueName: \"kubernetes.io/projected/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81-kube-api-access-2dtzm\") pod \"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81\" (UID: \"dc551cca-8f1e-4871-8f4b-3bf6a4be2b81\") "
Sep 15 09:33:55.827318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:55.827297    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81-kube-api-access-2dtzm" (OuterVolumeSpecName: "kube-api-access-2dtzm") pod "dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" (UID: "dc551cca-8f1e-4871-8f4b-3bf6a4be2b81"). InnerVolumeSpecName "kube-api-access-2dtzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:33:55.926281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:55.926183    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtzm\" (UniqueName: \"kubernetes.io/projected/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81-kube-api-access-2dtzm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:33:56.055961 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:56.055923    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" path="/var/lib/kubelet/pods/dc551cca-8f1e-4871-8f4b-3bf6a4be2b81/volumes"
Sep 15 09:33:56.621037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:56.620999    2571 generic.go:356] "Generic (PLEG): container finished" podID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" containerID="8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b" exitCode=137
Sep 15 09:33:56.621037 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:56.621038    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-14b71f"
Sep 15 09:33:56.621311 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:56.621101    2571 scope.go:117] "RemoveContainer" containerID="8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b"
Sep 15 09:33:56.637445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:56.637375    2571 scope.go:117] "RemoveContainer" containerID="8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b"
Sep 15 09:33:56.637850 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:33:56.637804    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b\": container with ID starting with 8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b not found: ID does not exist" containerID="8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b"
Sep 15 09:33:56.637951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:33:56.637864    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b"} err="failed to get container status \"8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b\": rpc error: code = NotFound desc = could not find container \"8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b\": container with ID starting with 8ff34ab48b78465efd6ed2a4463a267a5722ae54e3c1b3ccb1acdb5d0f24e23b not found: ID does not exist"
Sep 15 09:34:03.401843 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:03.401798    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:34:06.574133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:06.574093    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"]
Sep 15 09:34:06.595769 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:06.595739    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:07.811516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:07.811480    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:11.330006 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:11.329961    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:15.301957 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:15.301920    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:34:16.612201 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612167    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"]
Sep 15 09:34:16.612726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612703    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" containerName="test-curl-7741-14b71f"
Sep 15 09:34:16.612774 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612732    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" containerName="test-curl-7741-14b71f"
Sep 15 09:34:16.612774 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612766    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy"
Sep 15 09:34:16.612857 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612775    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy"
Sep 15 09:34:16.612897 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612886    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="56abe022-36c3-4004-91a8-ed166248ea47" containerName="istio-proxy"
Sep 15 09:34:16.612934 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.612905    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc551cca-8f1e-4871-8f4b-3bf6a4be2b81" containerName="test-curl-7741-14b71f"
Sep 15 09:34:16.617428 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.617404    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.620112 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.620088    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f79800\"/\"sub-test-model-f79800-kserve-self-signed-certs\""
Sep 15 09:34:16.625879 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.625852    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"]
Sep 15 09:34:16.631467 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.631440    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:16.735302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.735266    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-tmp-dir\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.735487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.735323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-dshm\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.735487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.735342    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-model-cache\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.735487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.735405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb1d391-705b-4821-bb01-d107a3740643-tls-certs\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.735487 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.735467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xxzq\" (UniqueName: \"kubernetes.io/projected/acb1d391-705b-4821-bb01-d107a3740643-kube-api-access-4xxzq\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.735651 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.735514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-home\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.836726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.836681    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-tmp-dir\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.836947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.836763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-dshm\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.836947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.836788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-model-cache\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.836947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.836813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb1d391-705b-4821-bb01-d107a3740643-tls-certs\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.836947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.836898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xxzq\" (UniqueName: \"kubernetes.io/projected/acb1d391-705b-4821-bb01-d107a3740643-kube-api-access-4xxzq\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.837178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.836975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-home\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.837178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.837132    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-tmp-dir\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.837285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.837180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-model-cache\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.837386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.837367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-home\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.839175 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.839156    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-dshm\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.839312 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.839295    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb1d391-705b-4821-bb01-d107a3740643-tls-certs\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.845901 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.845875    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xxzq\" (UniqueName: \"kubernetes.io/projected/acb1d391-705b-4821-bb01-d107a3740643-kube-api-access-4xxzq\") pod \"sub-test-model-f79800-kserve-5db57cf76c-kgb78\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") " pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:16.928266 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:16.928164    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:17.198912 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:17.198887    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"]
Sep 15 09:34:17.201063 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:34:17.201037    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacb1d391_705b_4821_bb01_d107a3740643.slice/crio-5b1e9bf56976176a105d36fcd0d08ff06b6375c50d129553bca57986a6314256 WatchSource:0}: Error finding container 5b1e9bf56976176a105d36fcd0d08ff06b6375c50d129553bca57986a6314256: Status 404 returned error can't find the container with id 5b1e9bf56976176a105d36fcd0d08ff06b6375c50d129553bca57986a6314256
Sep 15 09:34:17.736666 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:17.736623    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78" event={"ID":"acb1d391-705b-4821-bb01-d107a3740643","Type":"ContainerStarted","Data":"0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac"}
Sep 15 09:34:17.737082 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:17.736673    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78" event={"ID":"acb1d391-705b-4821-bb01-d107a3740643","Type":"ContainerStarted","Data":"5b1e9bf56976176a105d36fcd0d08ff06b6375c50d129553bca57986a6314256"}
Sep 15 09:34:17.761199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:17.756430    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78" podStartSLOduration=1.756408858 podStartE2EDuration="1.756408858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:34:17.753698812 +0000 UTC m=+1650.212988355" watchObservedRunningTime="2026-09-15 09:34:17.756408858 +0000 UTC m=+1650.215698400"
Sep 15 09:34:18.315945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:18.315913    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:18.417030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:18.416997    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:21.593322 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.593284    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj" podUID="0f260f8c-b7ab-4572-b603-5fdef8276330" containerName="main" containerID="cri-o://53183c5287879c310f7a1f44c546e8aa6d2a64068a14ff7c825970b73463ff20" gracePeriod=45
Sep 15 09:34:21.764033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.763801    2571 generic.go:356] "Generic (PLEG): container finished" podID="0f260f8c-b7ab-4572-b603-5fdef8276330" containerID="53183c5287879c310f7a1f44c546e8aa6d2a64068a14ff7c825970b73463ff20" exitCode=0
Sep 15 09:34:21.764033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.763985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj" event={"ID":"0f260f8c-b7ab-4572-b603-5fdef8276330","Type":"ContainerDied","Data":"53183c5287879c310f7a1f44c546e8aa6d2a64068a14ff7c825970b73463ff20"}
Sep 15 09:34:21.843275 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.843245    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:34:21.879290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879203    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-home\") pod \"0f260f8c-b7ab-4572-b603-5fdef8276330\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") "
Sep 15 09:34:21.879290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879255    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-dshm\") pod \"0f260f8c-b7ab-4572-b603-5fdef8276330\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") "
Sep 15 09:34:21.879290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879279    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-tmp-dir\") pod \"0f260f8c-b7ab-4572-b603-5fdef8276330\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") "
Sep 15 09:34:21.879565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879315    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f260f8c-b7ab-4572-b603-5fdef8276330-tls-certs\") pod \"0f260f8c-b7ab-4572-b603-5fdef8276330\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") "
Sep 15 09:34:21.879565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879376    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b47q\" (UniqueName: \"kubernetes.io/projected/0f260f8c-b7ab-4572-b603-5fdef8276330-kube-api-access-9b47q\") pod \"0f260f8c-b7ab-4572-b603-5fdef8276330\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") "
Sep 15 09:34:21.879565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879417    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-model-cache\") pod \"0f260f8c-b7ab-4572-b603-5fdef8276330\" (UID: \"0f260f8c-b7ab-4572-b603-5fdef8276330\") "
Sep 15 09:34:21.879565 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-home" (OuterVolumeSpecName: "home") pod "0f260f8c-b7ab-4572-b603-5fdef8276330" (UID: "0f260f8c-b7ab-4572-b603-5fdef8276330"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:21.879754 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f260f8c-b7ab-4572-b603-5fdef8276330" (UID: "0f260f8c-b7ab-4572-b603-5fdef8276330"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:21.879812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879751    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:34:21.879812 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879771    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:34:21.879932 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.879839    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f260f8c-b7ab-4572-b603-5fdef8276330" (UID: "0f260f8c-b7ab-4572-b603-5fdef8276330"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:21.881590 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.881562    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-dshm" (OuterVolumeSpecName: "dshm") pod "0f260f8c-b7ab-4572-b603-5fdef8276330" (UID: "0f260f8c-b7ab-4572-b603-5fdef8276330"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:34:21.881750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.881731    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f260f8c-b7ab-4572-b603-5fdef8276330-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f260f8c-b7ab-4572-b603-5fdef8276330" (UID: "0f260f8c-b7ab-4572-b603-5fdef8276330"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:34:21.881892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.881866    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f260f8c-b7ab-4572-b603-5fdef8276330-kube-api-access-9b47q" (OuterVolumeSpecName: "kube-api-access-9b47q") pod "0f260f8c-b7ab-4572-b603-5fdef8276330" (UID: "0f260f8c-b7ab-4572-b603-5fdef8276330"). InnerVolumeSpecName "kube-api-access-9b47q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:34:21.980944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.980900    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b47q\" (UniqueName: \"kubernetes.io/projected/0f260f8c-b7ab-4572-b603-5fdef8276330-kube-api-access-9b47q\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:34:21.980944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.980937    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:34:21.980944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.980950    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f260f8c-b7ab-4572-b603-5fdef8276330-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:34:21.981194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:21.980960    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f260f8c-b7ab-4572-b603-5fdef8276330-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:34:22.770661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:22.770621    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj" event={"ID":"0f260f8c-b7ab-4572-b603-5fdef8276330","Type":"ContainerDied","Data":"f11ad579aa23e6f7d4886ff69bc148ec80650f1b0d5b334cb2a216ff4d1f48cb"}
Sep 15 09:34:22.770661 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:22.770653    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"
Sep 15 09:34:22.771184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:22.770687    2571 scope.go:117] "RemoveContainer" containerID="53183c5287879c310f7a1f44c546e8aa6d2a64068a14ff7c825970b73463ff20"
Sep 15 09:34:22.787269 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:22.787238    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"]
Sep 15 09:34:22.790836 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:22.790801    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/auth-test-model-f79800-kserve-77b8b7b675-ljnxj"]
Sep 15 09:34:24.055414 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:24.055375    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f260f8c-b7ab-4572-b603-5fdef8276330" path="/var/lib/kubelet/pods/0f260f8c-b7ab-4572-b603-5fdef8276330/volumes"
Sep 15 09:34:26.159188 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:26.159143    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:26.508374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:26.508283    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:34:26.928695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:26.928649    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:26.928695 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:26.928708    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:26.941537 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:26.941503    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:27.809516 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:27.809430    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:34:43.792909 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.792861    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"]
Sep 15 09:34:43.793478 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.793459    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f260f8c-b7ab-4572-b603-5fdef8276330" containerName="main"
Sep 15 09:34:43.793523 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.793483    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f260f8c-b7ab-4572-b603-5fdef8276330" containerName="main"
Sep 15 09:34:43.793624 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.793612    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f260f8c-b7ab-4572-b603-5fdef8276330" containerName="main"
Sep 15 09:34:43.798492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.798465    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.802260 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.802237    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"openshift-service-ca.crt\""
Sep 15 09:34:43.802409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.802268    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"kube-root-ca.crt\""
Sep 15 09:34:43.802409 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.802269    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"sub-test-model-8dd3bd-kserve-self-signed-certs\""
Sep 15 09:34:43.802595 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.802578    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"default-dockercfg-6xhfl\""
Sep 15 09:34:43.807238 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.807213    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"]
Sep 15 09:34:43.826655 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.826625    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:43.892511 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.892469    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tls-certs\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.892704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.892517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tmp-dir\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.892704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.892551    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfw2s\" (UniqueName: \"kubernetes.io/projected/090dd568-bcc4-44c9-bc7d-64e84d4638bd-kube-api-access-hfw2s\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.892704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.892619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-home\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.892704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.892653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-dshm\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.892704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.892703    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-model-cache\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.993621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.993572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-home\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.993621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.993630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-dshm\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.993945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.993668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-model-cache\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.993945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.993737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tls-certs\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.993945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.993764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tmp-dir\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.993945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.993793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfw2s\" (UniqueName: \"kubernetes.io/projected/090dd568-bcc4-44c9-bc7d-64e84d4638bd-kube-api-access-hfw2s\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.994156 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.994016    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-home\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.994156 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.994115    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-model-cache\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.994304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.994166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tmp-dir\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.996045 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.996015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-dshm\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:43.996264 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:43.996246    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tls-certs\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:44.001983 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:44.001960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfw2s\" (UniqueName: \"kubernetes.io/projected/090dd568-bcc4-44c9-bc7d-64e84d4638bd-kube-api-access-hfw2s\") pod \"sub-test-model-8dd3bd-kserve-596d687cdf-tnphd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:44.110055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:44.110015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:44.383477 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:44.383445    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"]
Sep 15 09:34:44.386295 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:34:44.386256    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod090dd568_bcc4_44c9_bc7d_64e84d4638bd.slice/crio-44ec906af6d87f7bb96ed00611a19013bce8e8afa11e63621c52d0bec0b1d9ac WatchSource:0}: Error finding container 44ec906af6d87f7bb96ed00611a19013bce8e8afa11e63621c52d0bec0b1d9ac: Status 404 returned error can't find the container with id 44ec906af6d87f7bb96ed00611a19013bce8e8afa11e63621c52d0bec0b1d9ac
Sep 15 09:34:44.886206 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:44.886166    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd" event={"ID":"090dd568-bcc4-44c9-bc7d-64e84d4638bd","Type":"ContainerStarted","Data":"2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb"}
Sep 15 09:34:44.886711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:44.886215    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd" event={"ID":"090dd568-bcc4-44c9-bc7d-64e84d4638bd","Type":"ContainerStarted","Data":"44ec906af6d87f7bb96ed00611a19013bce8e8afa11e63621c52d0bec0b1d9ac"}
Sep 15 09:34:44.905376 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:44.905328    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd" podStartSLOduration=1.9053100349999998 podStartE2EDuration="1.905310035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:34:44.903859361 +0000 UTC m=+1677.363148897" watchObservedRunningTime="2026-09-15 09:34:44.905310035 +0000 UTC m=+1677.364599576"
Sep 15 09:34:47.308205 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:47.308170    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:34:54.110726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:54.110684    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:54.111233 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:54.110742    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:54.124486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:54.124454    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:54.945945 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:54.945908    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:34:58.627925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:58.627891    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:34:58.728204 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:34:58.728167    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:35:00.111155 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:00.111118    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:35:15.301571 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:15.301483    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:35:17.097903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:17.097855    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"]
Sep 15 09:35:17.122252 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:17.122209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:35:18.087049 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:18.087011    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"]
Sep 15 09:35:18.106521 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:18.106474    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:35:27.275071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:27.275023    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:35:27.523923 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:27.523895    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:35:27.602329 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:27.602287    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:35:32.117846 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.117754    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78" podUID="acb1d391-705b-4821-bb01-d107a3740643" containerName="main" containerID="cri-o://0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac" gracePeriod=45
Sep 15 09:35:32.389408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.389366    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:35:32.576152 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576116    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-home\") pod \"acb1d391-705b-4821-bb01-d107a3740643\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") "
Sep 15 09:35:32.576326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576169    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb1d391-705b-4821-bb01-d107a3740643-tls-certs\") pod \"acb1d391-705b-4821-bb01-d107a3740643\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") "
Sep 15 09:35:32.576326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576212    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-model-cache\") pod \"acb1d391-705b-4821-bb01-d107a3740643\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") "
Sep 15 09:35:32.576326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576246    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xxzq\" (UniqueName: \"kubernetes.io/projected/acb1d391-705b-4821-bb01-d107a3740643-kube-api-access-4xxzq\") pod \"acb1d391-705b-4821-bb01-d107a3740643\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") "
Sep 15 09:35:32.576326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576274    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-tmp-dir\") pod \"acb1d391-705b-4821-bb01-d107a3740643\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") "
Sep 15 09:35:32.576505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576414    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-home" (OuterVolumeSpecName: "home") pod "acb1d391-705b-4821-bb01-d107a3740643" (UID: "acb1d391-705b-4821-bb01-d107a3740643"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:32.576505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576430    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-dshm\") pod \"acb1d391-705b-4821-bb01-d107a3740643\" (UID: \"acb1d391-705b-4821-bb01-d107a3740643\") "
Sep 15 09:35:32.576607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576556    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-model-cache" (OuterVolumeSpecName: "model-cache") pod "acb1d391-705b-4821-bb01-d107a3740643" (UID: "acb1d391-705b-4821-bb01-d107a3740643"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:32.576607 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576567    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acb1d391-705b-4821-bb01-d107a3740643" (UID: "acb1d391-705b-4821-bb01-d107a3740643"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:32.576887 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576867    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:32.576887 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576887    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:32.577040 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.576897    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:32.578445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.578424    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-dshm" (OuterVolumeSpecName: "dshm") pod "acb1d391-705b-4821-bb01-d107a3740643" (UID: "acb1d391-705b-4821-bb01-d107a3740643"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:32.578544 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.578440    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb1d391-705b-4821-bb01-d107a3740643-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acb1d391-705b-4821-bb01-d107a3740643" (UID: "acb1d391-705b-4821-bb01-d107a3740643"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:35:32.578544 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.578468    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb1d391-705b-4821-bb01-d107a3740643-kube-api-access-4xxzq" (OuterVolumeSpecName: "kube-api-access-4xxzq") pod "acb1d391-705b-4821-bb01-d107a3740643" (UID: "acb1d391-705b-4821-bb01-d107a3740643"). InnerVolumeSpecName "kube-api-access-4xxzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:35:32.678372 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.678280    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xxzq\" (UniqueName: \"kubernetes.io/projected/acb1d391-705b-4821-bb01-d107a3740643-kube-api-access-4xxzq\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:32.678372 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.678310    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acb1d391-705b-4821-bb01-d107a3740643-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:32.678372 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:32.678320    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acb1d391-705b-4821-bb01-d107a3740643-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:33.107304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.107269    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd" podUID="090dd568-bcc4-44c9-bc7d-64e84d4638bd" containerName="main" containerID="cri-o://2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb" gracePeriod=45
Sep 15 09:35:33.135282 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.135251    2571 generic.go:356] "Generic (PLEG): container finished" podID="acb1d391-705b-4821-bb01-d107a3740643" containerID="0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac" exitCode=0
Sep 15 09:35:33.135697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.135292    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"
Sep 15 09:35:33.135697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.135333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78" event={"ID":"acb1d391-705b-4821-bb01-d107a3740643","Type":"ContainerDied","Data":"0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac"}
Sep 15 09:35:33.135697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.135378    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78" event={"ID":"acb1d391-705b-4821-bb01-d107a3740643","Type":"ContainerDied","Data":"5b1e9bf56976176a105d36fcd0d08ff06b6375c50d129553bca57986a6314256"}
Sep 15 09:35:33.135697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.135403    2571 scope.go:117] "RemoveContainer" containerID="0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac"
Sep 15 09:35:33.210171 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.210137    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"]
Sep 15 09:35:33.218405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.218375    2571 scope.go:117] "RemoveContainer" containerID="0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac"
Sep 15 09:35:33.218491 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.218381    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f79800/sub-test-model-f79800-kserve-5db57cf76c-kgb78"]
Sep 15 09:35:33.218787 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:35:33.218769    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac\": container with ID starting with 0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac not found: ID does not exist" containerID="0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac"
Sep 15 09:35:33.218875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.218796    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac"} err="failed to get container status \"0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac\": rpc error: code = NotFound desc = could not find container \"0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac\": container with ID starting with 0ec6c1c90c0cc5eefe868b6ed4edd687e53f7fabf3b80afba5dc6ebdb83a3bac not found: ID does not exist"
Sep 15 09:35:33.357930 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.357871    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:35:33.485259 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485216    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfw2s\" (UniqueName: \"kubernetes.io/projected/090dd568-bcc4-44c9-bc7d-64e84d4638bd-kube-api-access-hfw2s\") pod \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") "
Sep 15 09:35:33.485433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485295    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-dshm\") pod \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") "
Sep 15 09:35:33.485433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485326    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tmp-dir\") pod \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") "
Sep 15 09:35:33.485433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485346    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tls-certs\") pod \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") "
Sep 15 09:35:33.485433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485366    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-model-cache\") pod \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") "
Sep 15 09:35:33.485433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485394    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-home\") pod \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\" (UID: \"090dd568-bcc4-44c9-bc7d-64e84d4638bd\") "
Sep 15 09:35:33.485744 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "090dd568-bcc4-44c9-bc7d-64e84d4638bd" (UID: "090dd568-bcc4-44c9-bc7d-64e84d4638bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:33.485816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485701    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "090dd568-bcc4-44c9-bc7d-64e84d4638bd" (UID: "090dd568-bcc4-44c9-bc7d-64e84d4638bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:33.485816 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.485755    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-home" (OuterVolumeSpecName: "home") pod "090dd568-bcc4-44c9-bc7d-64e84d4638bd" (UID: "090dd568-bcc4-44c9-bc7d-64e84d4638bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:33.487555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.487524    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "090dd568-bcc4-44c9-bc7d-64e84d4638bd" (UID: "090dd568-bcc4-44c9-bc7d-64e84d4638bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:35:33.487659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.487574    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090dd568-bcc4-44c9-bc7d-64e84d4638bd-kube-api-access-hfw2s" (OuterVolumeSpecName: "kube-api-access-hfw2s") pod "090dd568-bcc4-44c9-bc7d-64e84d4638bd" (UID: "090dd568-bcc4-44c9-bc7d-64e84d4638bd"). InnerVolumeSpecName "kube-api-access-hfw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:35:33.487659 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.487588    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-dshm" (OuterVolumeSpecName: "dshm") pod "090dd568-bcc4-44c9-bc7d-64e84d4638bd" (UID: "090dd568-bcc4-44c9-bc7d-64e84d4638bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:35:33.586481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.586424    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfw2s\" (UniqueName: \"kubernetes.io/projected/090dd568-bcc4-44c9-bc7d-64e84d4638bd-kube-api-access-hfw2s\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:33.586481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.586473    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:33.586481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.586484    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:33.586481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.586492    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/090dd568-bcc4-44c9-bc7d-64e84d4638bd-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:33.586481 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.586501    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:33.586801 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:33.586508    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/090dd568-bcc4-44c9-bc7d-64e84d4638bd-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:35:34.055107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.055071    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb1d391-705b-4821-bb01-d107a3740643" path="/var/lib/kubelet/pods/acb1d391-705b-4821-bb01-d107a3740643/volumes"
Sep 15 09:35:34.142060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.142011    2571 generic.go:356] "Generic (PLEG): container finished" podID="090dd568-bcc4-44c9-bc7d-64e84d4638bd" containerID="2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb" exitCode=0
Sep 15 09:35:34.142582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.142067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd" event={"ID":"090dd568-bcc4-44c9-bc7d-64e84d4638bd","Type":"ContainerDied","Data":"2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb"}
Sep 15 09:35:34.142582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.142116    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd" event={"ID":"090dd568-bcc4-44c9-bc7d-64e84d4638bd","Type":"ContainerDied","Data":"44ec906af6d87f7bb96ed00611a19013bce8e8afa11e63621c52d0bec0b1d9ac"}
Sep 15 09:35:34.142582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.142117    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"
Sep 15 09:35:34.142582 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.142134    2571 scope.go:117] "RemoveContainer" containerID="2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb"
Sep 15 09:35:34.150936 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.150917    2571 scope.go:117] "RemoveContainer" containerID="2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb"
Sep 15 09:35:34.151293 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:35:34.151267    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb\": container with ID starting with 2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb not found: ID does not exist" containerID="2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb"
Sep 15 09:35:34.151384 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.151305    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb"} err="failed to get container status \"2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb\": rpc error: code = NotFound desc = could not find container \"2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb\": container with ID starting with 2551b190bc031f4d95eb50fbca7d7e4de09e47fb6264e71227cbec8fcc8b5ceb not found: ID does not exist"
Sep 15 09:35:34.174167 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.174139    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"]
Sep 15 09:35:34.175622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:34.175600    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/sub-test-model-8dd3bd-kserve-596d687cdf-tnphd"]
Sep 15 09:35:36.055033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:36.055000    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090dd568-bcc4-44c9-bc7d-64e84d4638bd" path="/var/lib/kubelet/pods/090dd568-bcc4-44c9-bc7d-64e84d4638bd/volumes"
Sep 15 09:35:36.301424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:36.301389    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:35:52.309728 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.309692    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"]
Sep 15 09:35:52.310222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.310084    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb1d391-705b-4821-bb01-d107a3740643" containerName="main"
Sep 15 09:35:52.310222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.310096    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb1d391-705b-4821-bb01-d107a3740643" containerName="main"
Sep 15 09:35:52.310222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.310105    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090dd568-bcc4-44c9-bc7d-64e84d4638bd" containerName="main"
Sep 15 09:35:52.310222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.310111    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="090dd568-bcc4-44c9-bc7d-64e84d4638bd" containerName="main"
Sep 15 09:35:52.310222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.310168    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb1d391-705b-4821-bb01-d107a3740643" containerName="main"
Sep 15 09:35:52.310222 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.310178    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="090dd568-bcc4-44c9-bc7d-64e84d4638bd" containerName="main"
Sep 15 09:35:52.312896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.312875    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.316553 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.316532    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"openshift-service-ca.crt\""
Sep 15 09:35:52.316553 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.316547    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"default-dockercfg-6xhfl\""
Sep 15 09:35:52.316777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.316581    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"kube-root-ca.crt\""
Sep 15 09:35:52.316777 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.316549    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8dd3bd\"/\"rate-test-model-8dd3bd-kserve-self-signed-certs\""
Sep 15 09:35:52.324982 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.324954    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"]
Sep 15 09:35:52.348819 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.348783    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:35:52.364851 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.364799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-home\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.365020 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.364861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-dshm\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.365020 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.364884    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c630a3-3881-4856-a6c5-e157fe694a20-tls-certs\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.365020 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.364907    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t98v\" (UniqueName: \"kubernetes.io/projected/20c630a3-3881-4856-a6c5-e157fe694a20-kube-api-access-8t98v\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.365020 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.364924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-tmp-dir\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.365160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.365041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-model-cache\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.465730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.465695    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-home\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.465947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.465765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-dshm\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.465947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.465794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c630a3-3881-4856-a6c5-e157fe694a20-tls-certs\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.465947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.465820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t98v\" (UniqueName: \"kubernetes.io/projected/20c630a3-3881-4856-a6c5-e157fe694a20-kube-api-access-8t98v\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.465947 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.465872    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-tmp-dir\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.466165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.465961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-model-cache\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.466165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.466124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-home\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.466285 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.466261    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-model-cache\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.466326 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.466285    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-tmp-dir\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.468107 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.468081    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-dshm\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.468258 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.468242    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c630a3-3881-4856-a6c5-e157fe694a20-tls-certs\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.475302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.475280    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t98v\" (UniqueName: \"kubernetes.io/projected/20c630a3-3881-4856-a6c5-e157fe694a20-kube-api-access-8t98v\") pod \"rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") " pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.625906 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.625870    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:35:52.895603 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:52.895558    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"]
Sep 15 09:35:52.897975 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:35:52.897939    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20c630a3_3881_4856_a6c5_e157fe694a20.slice/crio-d864557343acf75d722180720ddaebcc83ffc5ce66d1c28cd0281c43605004d9 WatchSource:0}: Error finding container d864557343acf75d722180720ddaebcc83ffc5ce66d1c28cd0281c43605004d9: Status 404 returned error can't find the container with id d864557343acf75d722180720ddaebcc83ffc5ce66d1c28cd0281c43605004d9
Sep 15 09:35:53.239863 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:53.239747    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh" event={"ID":"20c630a3-3881-4856-a6c5-e157fe694a20","Type":"ContainerStarted","Data":"45f85cc733e7d2ca7360205768e73d0c21c421d40688c1184229dfa0b4c057da"}
Sep 15 09:35:53.239863 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:53.239794    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh" event={"ID":"20c630a3-3881-4856-a6c5-e157fe694a20","Type":"ContainerStarted","Data":"d864557343acf75d722180720ddaebcc83ffc5ce66d1c28cd0281c43605004d9"}
Sep 15 09:35:53.285160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:53.285104    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh" podStartSLOduration=1.285085039 podStartE2EDuration="1.285085039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:35:53.282996056 +0000 UTC m=+1745.742285597" watchObservedRunningTime="2026-09-15 09:35:53.285085039 +0000 UTC m=+1745.744374580"
Sep 15 09:35:56.007413 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:35:56.007366    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:36:02.626896 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:02.626857    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:36:02.627461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:02.626925    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:36:02.639876 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:02.639818    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:36:03.302309 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:03.302269    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:36:12.003133 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:12.003092    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:36:21.606351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:21.606314    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:36:33.795522 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:33.795484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:33.805475 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:33.805440    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:36:38.403030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:38.402998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:39.418931 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:39.418895    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"]
Sep 15 09:36:39.435455 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:39.435428    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:39.846420 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:39.846391    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:40.121521 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.121396    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"]
Sep 15 09:36:40.125148 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.125122    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.127576 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.127549    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-08aa53-openshift-default-dockercfg-6xqrr\""
Sep 15 09:36:40.137668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.137640    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"]
Sep 15 09:36:40.152586 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.152550    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:40.234795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.234754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-credential-socket\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.234799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-envoy\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.234877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdtj\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-kube-api-access-vxdtj\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.234901    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-token\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.234949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-data\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.234968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd110e9-fcc1-4830-940c-7c8bbe919831-istiod-ca-cert\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.235019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-podinfo\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.235048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-certs\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.235407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.235067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-socket\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.252844 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.252804    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:40.336583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336542    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-data\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336586    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd110e9-fcc1-4830-940c-7c8bbe919831-istiod-ca-cert\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336627    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-podinfo\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-certs\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-socket\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-credential-socket\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-envoy\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdtj\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-kube-api-access-vxdtj\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.336903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.336870    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-token\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.337287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.337100    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-data\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.337287 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.337143    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-socket\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.337408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.337334    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-certs\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.337408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.337322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-credential-socket\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.337564 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.337543    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd110e9-fcc1-4830-940c-7c8bbe919831-istiod-ca-cert\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.339130 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.339109    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-envoy\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.339347 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.339329    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-podinfo\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.345126 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.345096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-token\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.345297 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.345278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdtj\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-kube-api-access-vxdtj\") pod \"e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") " pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.438202 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.438102    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:40.715672 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.715624    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"]
Sep 15 09:36:40.718269 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:36:40.718228    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd110e9_fcc1_4830_940c_7c8bbe919831.slice/crio-b4396c11e9d43516c47a0299e96a81b144607aea143e2065a224c0d8771a7c05 WatchSource:0}: Error finding container b4396c11e9d43516c47a0299e96a81b144607aea143e2065a224c0d8771a7c05: Status 404 returned error can't find the container with id b4396c11e9d43516c47a0299e96a81b144607aea143e2065a224c0d8771a7c05
Sep 15 09:36:40.720764 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.720727    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:36:40.720910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.720803    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:36:40.720910 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:40.720857    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:36:41.485673 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:41.485630    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" event={"ID":"8bd110e9-fcc1-4830-940c-7c8bbe919831","Type":"ContainerStarted","Data":"710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"}
Sep 15 09:36:41.486117 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:41.485735    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" event={"ID":"8bd110e9-fcc1-4830-940c-7c8bbe919831","Type":"ContainerStarted","Data":"b4396c11e9d43516c47a0299e96a81b144607aea143e2065a224c0d8771a7c05"}
Sep 15 09:36:41.506668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:41.506617    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podStartSLOduration=1.5066006600000001 podStartE2EDuration="1.50660066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:36:41.505018718 +0000 UTC m=+1793.964308260" watchObservedRunningTime="2026-09-15 09:36:41.50660066 +0000 UTC m=+1793.965890200"
Sep 15 09:36:42.438526 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:42.438484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:42.443875 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:42.443850    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:42.491101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:42.491060    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:42.492099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:42.492073    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:36:43.182422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:43.182383    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:43.541716 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:43.541620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:43.641593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:43.641546    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:43.947408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:43.947350    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:44.111926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.111892    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:36:44.862936 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.862898    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"]
Sep 15 09:36:44.866669 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.866649    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:44.870120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.870082    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 09:36:44.870290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.870268    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 09:36:44.870958 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.870940    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ptrgq\""
Sep 15 09:36:44.871046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.870945    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-08aa53-serving-cert\""
Sep 15 09:36:44.871046 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.870977    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 09:36:44.878591 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.878138    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"]
Sep 15 09:36:44.909951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.909918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:44.982638 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.982604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4vw\" (UniqueName: \"kubernetes.io/projected/8f770217-796e-4168-aa18-d3fc456249cc-kube-api-access-cr4vw\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:44.982854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.982737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f770217-796e-4168-aa18-d3fc456249cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:44.982854 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.982771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-metrics-certs\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:44.982950 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:44.982855    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.007024 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.006988    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:45.083958 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.083922    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4vw\" (UniqueName: \"kubernetes.io/projected/8f770217-796e-4168-aa18-d3fc456249cc-kube-api-access-cr4vw\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.084185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.083999    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f770217-796e-4168-aa18-d3fc456249cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.084185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.084024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-metrics-certs\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.084185 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.084074    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.084801 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.084776    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f770217-796e-4168-aa18-d3fc456249cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.086608 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.086586    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.086699 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.086652    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-metrics-certs\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.100554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.100513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4vw\" (UniqueName: \"kubernetes.io/projected/8f770217-796e-4168-aa18-d3fc456249cc-kube-api-access-cr4vw\") pod \"maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.133654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.133573    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:45.180060 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.180020    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:45.455371 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.455345    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"]
Sep 15 09:36:45.457404 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:36:45.457372    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f770217_796e_4168_aa18_d3fc456249cc.slice/crio-c14c6c68c91cf16be1f3a34274c51d9452df3f4fff3db4aac09b58318e90beec WatchSource:0}: Error finding container c14c6c68c91cf16be1f3a34274c51d9452df3f4fff3db4aac09b58318e90beec: Status 404 returned error can't find the container with id c14c6c68c91cf16be1f3a34274c51d9452df3f4fff3db4aac09b58318e90beec
Sep 15 09:36:45.459305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.459279    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:36:45.502288 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.502221    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:45.507163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.507129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4" event={"ID":"8f770217-796e-4168-aa18-d3fc456249cc","Type":"ContainerStarted","Data":"c14c6c68c91cf16be1f3a34274c51d9452df3f4fff3db4aac09b58318e90beec"}
Sep 15 09:36:45.598453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:45.598396    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:46.515366 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:46.515324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4" event={"ID":"8f770217-796e-4168-aa18-d3fc456249cc","Type":"ContainerStarted","Data":"c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382"}
Sep 15 09:36:46.515776 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:46.515421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:46.535247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:46.535142    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4" podStartSLOduration=2.137946294 podStartE2EDuration="2.535127167s" totalImagesPullingTime="397.180873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:36:46.533619661 +0000 UTC m=+1798.992909203" watchObservedRunningTime="2026-09-15 09:36:46.535127167 +0000 UTC m=+1798.994416708"
Sep 15 09:36:48.116714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:48.116675    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:36:48.122998 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:48.122972    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:36:48.301181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:48.301147    2571 scope.go:117] "RemoveContainer" containerID="0ff415d8b92cc450397116623a2f40f31a5e98b60853cd07418f1babaf337007"
Sep 15 09:36:48.309767 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:48.309743    2571 scope.go:117] "RemoveContainer" containerID="a49fd3276ef5eb091fa8b6c8514f3a34a1ed6a0aec6ac5dd82f995f06efb2992"
Sep 15 09:36:48.318161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:48.318127    2571 scope.go:117] "RemoveContainer" containerID="9e4fb8a446310bf02869cde3a9c4db35911e20a58e596d7f934a48ae0ae6be52"
Sep 15 09:36:52.535213 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:52.535166    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:36:54.443668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.443580    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh" podUID="20c630a3-3881-4856-a6c5-e157fe694a20" containerName="main" containerID="cri-o://45f85cc733e7d2ca7360205768e73d0c21c421d40688c1184229dfa0b4c057da" gracePeriod=45
Sep 15 09:36:54.564382 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.564353    2571 generic.go:356] "Generic (PLEG): container finished" podID="20c630a3-3881-4856-a6c5-e157fe694a20" containerID="45f85cc733e7d2ca7360205768e73d0c21c421d40688c1184229dfa0b4c057da" exitCode=0
Sep 15 09:36:54.564500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.564433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh" event={"ID":"20c630a3-3881-4856-a6c5-e157fe694a20","Type":"ContainerDied","Data":"45f85cc733e7d2ca7360205768e73d0c21c421d40688c1184229dfa0b4c057da"}
Sep 15 09:36:54.601381 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.601346    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:36:54.715123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.715095    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:36:54.780336 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780302    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-model-cache\") pod \"20c630a3-3881-4856-a6c5-e157fe694a20\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") "
Sep 15 09:36:54.780502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-home\") pod \"20c630a3-3881-4856-a6c5-e157fe694a20\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") "
Sep 15 09:36:54.780502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780394    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c630a3-3881-4856-a6c5-e157fe694a20-tls-certs\") pod \"20c630a3-3881-4856-a6c5-e157fe694a20\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") "
Sep 15 09:36:54.780502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780425    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-tmp-dir\") pod \"20c630a3-3881-4856-a6c5-e157fe694a20\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") "
Sep 15 09:36:54.780502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780483    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-dshm\") pod \"20c630a3-3881-4856-a6c5-e157fe694a20\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") "
Sep 15 09:36:54.780710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780526    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t98v\" (UniqueName: \"kubernetes.io/projected/20c630a3-3881-4856-a6c5-e157fe694a20-kube-api-access-8t98v\") pod \"20c630a3-3881-4856-a6c5-e157fe694a20\" (UID: \"20c630a3-3881-4856-a6c5-e157fe694a20\") "
Sep 15 09:36:54.780710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780637    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-model-cache" (OuterVolumeSpecName: "model-cache") pod "20c630a3-3881-4856-a6c5-e157fe694a20" (UID: "20c630a3-3881-4856-a6c5-e157fe694a20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:54.780710 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780656    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-home" (OuterVolumeSpecName: "home") pod "20c630a3-3881-4856-a6c5-e157fe694a20" (UID: "20c630a3-3881-4856-a6c5-e157fe694a20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:54.780857 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20c630a3-3881-4856-a6c5-e157fe694a20" (UID: "20c630a3-3881-4856-a6c5-e157fe694a20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:54.780952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780924    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:36:54.780952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780944    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:36:54.780952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.780952    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:36:54.782542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.782519    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c630a3-3881-4856-a6c5-e157fe694a20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20c630a3-3881-4856-a6c5-e157fe694a20" (UID: "20c630a3-3881-4856-a6c5-e157fe694a20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:36:54.782542 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.782522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-dshm" (OuterVolumeSpecName: "dshm") pod "20c630a3-3881-4856-a6c5-e157fe694a20" (UID: "20c630a3-3881-4856-a6c5-e157fe694a20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:36:54.782664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.782625    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c630a3-3881-4856-a6c5-e157fe694a20-kube-api-access-8t98v" (OuterVolumeSpecName: "kube-api-access-8t98v") pod "20c630a3-3881-4856-a6c5-e157fe694a20" (UID: "20c630a3-3881-4856-a6c5-e157fe694a20"). InnerVolumeSpecName "kube-api-access-8t98v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:36:54.882348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.882314    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c630a3-3881-4856-a6c5-e157fe694a20-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:36:54.882348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.882343    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c630a3-3881-4856-a6c5-e157fe694a20-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:36:54.882348 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:54.882353    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t98v\" (UniqueName: \"kubernetes.io/projected/20c630a3-3881-4856-a6c5-e157fe694a20-kube-api-access-8t98v\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:36:55.579321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:55.579276    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh" event={"ID":"20c630a3-3881-4856-a6c5-e157fe694a20","Type":"ContainerDied","Data":"d864557343acf75d722180720ddaebcc83ffc5ce66d1c28cd0281c43605004d9"}
Sep 15 09:36:55.579321 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:55.579308    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"
Sep 15 09:36:55.579857 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:55.579343    2571 scope.go:117] "RemoveContainer" containerID="45f85cc733e7d2ca7360205768e73d0c21c421d40688c1184229dfa0b4c057da"
Sep 15 09:36:55.599868 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:55.599822    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"]
Sep 15 09:36:55.603535 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:55.603507    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8dd3bd/rate-test-model-8dd3bd-kserve-b6bd978b7-kwpkh"]
Sep 15 09:36:55.769480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:55.769452    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:56.055591 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:56.055551    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c630a3-3881-4856-a6c5-e157fe694a20" path="/var/lib/kubelet/pods/20c630a3-3881-4856-a6c5-e157fe694a20/volumes"
Sep 15 09:36:58.534005 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:58.533966    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 09:36:58.534411 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:58.534047    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 09:36:59.541468 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.541431    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:36:59.601350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.601321    2571 generic.go:356] "Generic (PLEG): container finished" podID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerID="710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91" exitCode=0
Sep 15 09:36:59.601503 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.601400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" event={"ID":"8bd110e9-fcc1-4830-940c-7c8bbe919831","Type":"ContainerDied","Data":"710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"}
Sep 15 09:36:59.601935 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.601907    2571 scope.go:117] "RemoveContainer" containerID="710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"
Sep 15 09:36:59.603095 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.603069    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:36:59.603161 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.603138    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:36:59.603202 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.603170    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:36:59.624759 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:36:59.624735    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:00.609034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:00.608999    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" event={"ID":"8bd110e9-fcc1-4830-940c-7c8bbe919831","Type":"ContainerStarted","Data":"60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255"}
Sep 15 09:37:01.438922 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:01.438882    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:37:01.439226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:01.439023    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 09:37:01.439396 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:01.439372    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 09:37:02.439317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:02.439275    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 09:37:02.439934 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:02.439898    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 09:37:03.439992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:03.439472    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 09:37:03.439992 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:03.439545    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 09:37:04.439556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:04.439514    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 09:37:04.439889 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:04.439854    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 09:37:05.439286 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:05.438794    2571 patch_prober.go:28] interesting pod/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 15 09:37:05.439286 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:05.438883    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 15 09:37:05.970042 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:05.970008    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:06.443042 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:06.443011    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:37:06.443422 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:06.443268    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:37:06.443919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:06.443902    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:37:07.194905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:07.194871    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:07.968753 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:07.968719    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:08.421114 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:08.421076    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:08.679072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:08.678983    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:09.066290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.066203    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:09.336770 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.336733    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"]
Sep 15 09:37:09.337388 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.337371    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c630a3-3881-4856-a6c5-e157fe694a20" containerName="main"
Sep 15 09:37:09.337452 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.337392    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c630a3-3881-4856-a6c5-e157fe694a20" containerName="main"
Sep 15 09:37:09.337518 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.337507    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c630a3-3881-4856-a6c5-e157fe694a20" containerName="main"
Sep 15 09:37:09.341099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.341077    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.344163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.344141    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6920e5-openshift-default-dockercfg-hnhmr\""
Sep 15 09:37:09.354781 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.354752    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"]
Sep 15 09:37:09.375191 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.375157    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:09.427424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istiod-ca-cert\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-certs\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427486    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-envoy\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-credential-socket\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427601 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-data\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427621    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-socket\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-podinfo\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-token\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.427743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.427710    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f44j8\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-kube-api-access-f44j8\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.474660 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.474622    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:09.529170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-data\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-socket\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529199    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-podinfo\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-token\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f44j8\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-kube-api-access-f44j8\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istiod-ca-cert\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-certs\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-envoy\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-credential-socket\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529613    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-data\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.529665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529631    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-socket\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.530003 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-certs\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.530003 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.529921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-credential-socket\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.530268 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.530244    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istiod-ca-cert\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.531658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.531640    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-envoy\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.531860 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.531815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-podinfo\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.537898 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.537872    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-token\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.538023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.537955    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f44j8\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-kube-api-access-f44j8\") pod \"e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") " pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.652989 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.652892    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:09.656184 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.656156    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:09.928748 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.928719    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"]
Sep 15 09:37:09.931262 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:37:09.931234    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1010dd_f739_4cb4_a0d6_38b0ffd8cbe9.slice/crio-bc7c8962757ebb0f5b2c9c752cf76b93188b3f3b1f6aa720544801700c023517 WatchSource:0}: Error finding container bc7c8962757ebb0f5b2c9c752cf76b93188b3f3b1f6aa720544801700c023517: Status 404 returned error can't find the container with id bc7c8962757ebb0f5b2c9c752cf76b93188b3f3b1f6aa720544801700c023517
Sep 15 09:37:09.933568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.933527    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:37:09.933674 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.933622    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:37:09.933717 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:09.933672    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:37:10.673250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:10.673210    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" event={"ID":"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9","Type":"ContainerStarted","Data":"881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"}
Sep 15 09:37:10.673715 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:10.673270    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" event={"ID":"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9","Type":"ContainerStarted","Data":"bc7c8962757ebb0f5b2c9c752cf76b93188b3f3b1f6aa720544801700c023517"}
Sep 15 09:37:10.692176 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:10.692117    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podStartSLOduration=1.692098905 podStartE2EDuration="1.692098905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:37:10.691536304 +0000 UTC m=+1823.150825857" watchObservedRunningTime="2026-09-15 09:37:10.692098905 +0000 UTC m=+1823.151388447"
Sep 15 09:37:11.653887 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:11.653848    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:11.658931 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:11.658905    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:11.678451 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:11.678418    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:11.679562 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:11.679541    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:11.976138 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:11.976048    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:12.573273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:12.573238    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:12.673701 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:12.673662    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:12.774226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:12.774191    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:13.600357 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:13.600319    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:14.356432 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:14.356388    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:15.058500 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:15.058455    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:15.144593 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:15.144556    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:15.519293 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:15.519256    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:15.622028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:15.621991    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:15.688968 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:15.688936    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:20.110842 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:20.110804    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:37:26.778489 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:26.778454    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:30.783921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:30.783885    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerID="881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130" exitCode=0
Sep 15 09:37:30.784358 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:30.783947    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" event={"ID":"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9","Type":"ContainerDied","Data":"881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"}
Sep 15 09:37:30.784430 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:30.784413    2571 scope.go:117] "RemoveContainer" containerID="881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"
Sep 15 09:37:30.785663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:30.785628    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:37:30.785776 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:30.785735    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:37:30.785860 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:30.785784    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 09:37:31.791690 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:31.791650    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" event={"ID":"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9","Type":"ContainerStarted","Data":"1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111"}
Sep 15 09:37:32.653861 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:32.653816    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:32.654078 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:32.654055    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 09:37:32.654152 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:32.654111    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 09:37:33.653644 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:33.653602    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 09:37:33.654067 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:33.653661    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 09:37:34.653971 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:34.653930    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 09:37:34.654368 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:34.653995    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 09:37:35.653541 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:35.653498    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 09:37:35.653729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:35.653574    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 09:37:36.031449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:36.031355    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:36.654136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:36.654096    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 15 09:37:36.654445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:36.654416    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 15 09:37:37.422713 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:37.422681    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:37.574882 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:37.574817    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:37.657365 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:37.657334    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:37.657639 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:37.657619    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:37.658323 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:37.658304    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:37:38.350794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:38.350762    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:39.014437 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:39.014403    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:39.687328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:39.687294    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:40.487618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:40.487578    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:47.303585 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.303552    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"]
Sep 15 09:37:47.308267 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.308243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.310931 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.310907    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-08aa53\"/\"openshift-service-ca.crt\""
Sep 15 09:37:47.311145 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.311131    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-08aa53\"/\"kube-root-ca.crt\""
Sep 15 09:37:47.312397 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.312379    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-08aa53\"/\"default-dockercfg-hwkhs\""
Sep 15 09:37:47.312472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.312412    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-08aa53\"/\"ipp-route-08aa53-kserve-self-signed-certs\""
Sep 15 09:37:47.316926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.316903    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"]
Sep 15 09:37:47.332622 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.332568    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:37:47.404283 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.404239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062dfa28-2363-44a6-bc35-b7e683807b95-tls-certs\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.404486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.404300    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-model-cache\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.404486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.404350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-home\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.404486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.404391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-tmp-dir\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.404486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.404428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jcs\" (UniqueName: \"kubernetes.io/projected/062dfa28-2363-44a6-bc35-b7e683807b95-kube-api-access-w5jcs\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.404711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.404504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-dshm\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506109 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.505965    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062dfa28-2363-44a6-bc35-b7e683807b95-tls-certs\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.506207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-model-cache\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.506260    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-home\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506442 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.506328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-tmp-dir\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506442 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.506393    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jcs\" (UniqueName: \"kubernetes.io/projected/062dfa28-2363-44a6-bc35-b7e683807b95-kube-api-access-w5jcs\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506442 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.506432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-dshm\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.506779 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.506753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-model-cache\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.507354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.507022    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-home\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.507354 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.507083    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-tmp-dir\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.509502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.509478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062dfa28-2363-44a6-bc35-b7e683807b95-tls-certs\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.509700 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.509678    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-dshm\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.517091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.517063    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jcs\" (UniqueName: \"kubernetes.io/projected/062dfa28-2363-44a6-bc35-b7e683807b95-kube-api-access-w5jcs\") pod \"ipp-route-08aa53-kserve-55686c8887-bclph\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") " pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.618578 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.618542    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:47.895216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:47.895191    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"]
Sep 15 09:37:47.897559 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:37:47.897523    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod062dfa28_2363_44a6_bc35_b7e683807b95.slice/crio-4df943600d5ed52495184ec875c97434023005141b7c9a199930b774a1543f1d WatchSource:0}: Error finding container 4df943600d5ed52495184ec875c97434023005141b7c9a199930b774a1543f1d: Status 404 returned error can't find the container with id 4df943600d5ed52495184ec875c97434023005141b7c9a199930b774a1543f1d
Sep 15 09:37:48.878680 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:48.878638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph" event={"ID":"062dfa28-2363-44a6-bc35-b7e683807b95","Type":"ContainerStarted","Data":"08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7"}
Sep 15 09:37:48.879091 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:48.878692    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph" event={"ID":"062dfa28-2363-44a6-bc35-b7e683807b95","Type":"ContainerStarted","Data":"4df943600d5ed52495184ec875c97434023005141b7c9a199930b774a1543f1d"}
Sep 15 09:37:48.895894 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:48.895843    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph" podStartSLOduration=1.8958096819999999 podStartE2EDuration="1.895809682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:37:48.895169851 +0000 UTC m=+1861.354459393" watchObservedRunningTime="2026-09-15 09:37:48.895809682 +0000 UTC m=+1861.355099216"
Sep 15 09:37:51.310142 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:51.310105    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:37:57.619198 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:57.619142    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:57.619874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:57.619215    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:57.632036 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:57.632004    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:37:57.934764 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:37:57.934677    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:38:12.007892 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:12.007795    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:38:18.823082 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:18.823041    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:19.161611 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:19.161579    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:19.317647 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:19.317609    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:21.775621 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:21.775585    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:21.898869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:21.898810    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:21.996952 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:21.996918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:22.602480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:22.602444    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:23.490087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:23.490043    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:24.171573 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:24.171545    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:24.923886 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:24.923850    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:25.024124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:25.024089    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:25.232495 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:25.232405    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:25.357408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:25.357371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:25.454320 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:25.454288    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:32.321469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:32.321426    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:38:46.836100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:46.836063    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:47.070619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:47.070584    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:48.184318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:48.184282    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:49.029488 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:49.029449    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:49.842740 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:49.842704    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:50.595604 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:50.595575    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:51.269048 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:51.269013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:52.083186 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:52.083154    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:53.220071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:53.220040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:53.817804 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:53.817763    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:53.915027 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:53.914997    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:54.015973 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:54.015938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:54.094017 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:54.093972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:54.157163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:54.157131    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:54.393703 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:54.393614    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:54.497096 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:54.497054    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:55.178346 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:55.178296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:55.877061 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:55.877030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:56.789160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:56.789128    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:57.455136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:57.455103    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:38:58.807401 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:38:58.807368    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:39:00.132017 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:00.131978    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:00.191226 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:00.191189    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:00.445460 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:00.445370    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:04.191449 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:04.191416    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:04.897568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:04.897533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:05.694907 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:05.694872    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:06.492569 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:06.492526    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:07.173531 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:07.173501    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:07.970442 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:07.970407    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:17.209795 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:17.209753    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:39:19.335988 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.335946    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"]
Sep 15 09:39:19.336472 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.336193    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4" podUID="8f770217-796e-4168-aa18-d3fc456249cc" containerName="maas-api" containerID="cri-o://c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382" gracePeriod=30
Sep 15 09:39:19.363131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.363101    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:19.436314 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.436275    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:19.496708 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.496669    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:19.578722 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.578694    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:19.602122 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.602095    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:39:19.637618 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.637583    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:19.738216 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.738183    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f770217-796e-4168-aa18-d3fc456249cc-openshift-service-ca\") pod \"8f770217-796e-4168-aa18-d3fc456249cc\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") "
Sep 15 09:39:19.738563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.738540    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-maas-api-tls\") pod \"8f770217-796e-4168-aa18-d3fc456249cc\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") "
Sep 15 09:39:19.738794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.738620    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f770217-796e-4168-aa18-d3fc456249cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f770217-796e-4168-aa18-d3fc456249cc" (UID: "8f770217-796e-4168-aa18-d3fc456249cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:39:19.739201 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.739183    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr4vw\" (UniqueName: \"kubernetes.io/projected/8f770217-796e-4168-aa18-d3fc456249cc-kube-api-access-cr4vw\") pod \"8f770217-796e-4168-aa18-d3fc456249cc\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") "
Sep 15 09:39:19.739400 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.739385    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-metrics-certs\") pod \"8f770217-796e-4168-aa18-d3fc456249cc\" (UID: \"8f770217-796e-4168-aa18-d3fc456249cc\") "
Sep 15 09:39:19.739865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.739843    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f770217-796e-4168-aa18-d3fc456249cc-openshift-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:19.741630 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.741584    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f770217-796e-4168-aa18-d3fc456249cc" (UID: "8f770217-796e-4168-aa18-d3fc456249cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:39:19.744369 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.744333    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f770217-796e-4168-aa18-d3fc456249cc-kube-api-access-cr4vw" (OuterVolumeSpecName: "kube-api-access-cr4vw") pod "8f770217-796e-4168-aa18-d3fc456249cc" (UID: "8f770217-796e-4168-aa18-d3fc456249cc"). InnerVolumeSpecName "kube-api-access-cr4vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:39:19.744575 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.744554    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8f770217-796e-4168-aa18-d3fc456249cc" (UID: "8f770217-796e-4168-aa18-d3fc456249cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:39:19.841066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.840969    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-maas-api-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:19.841066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.841003    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4vw\" (UniqueName: \"kubernetes.io/projected/8f770217-796e-4168-aa18-d3fc456249cc-kube-api-access-cr4vw\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:19.841066 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.841015    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f770217-796e-4168-aa18-d3fc456249cc-metrics-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:19.931269 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:19.931233    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:20.028560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.028518    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:20.344987 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.344950    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f770217-796e-4168-aa18-d3fc456249cc" containerID="c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382" exitCode=0
Sep 15 09:39:20.345469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.344994    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4" event={"ID":"8f770217-796e-4168-aa18-d3fc456249cc","Type":"ContainerDied","Data":"c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382"}
Sep 15 09:39:20.345469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.344999    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"
Sep 15 09:39:20.345469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.345041    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4" event={"ID":"8f770217-796e-4168-aa18-d3fc456249cc","Type":"ContainerDied","Data":"c14c6c68c91cf16be1f3a34274c51d9452df3f4fff3db4aac09b58318e90beec"}
Sep 15 09:39:20.345469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.345067    2571 scope.go:117] "RemoveContainer" containerID="c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382"
Sep 15 09:39:20.354881 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.354861    2571 scope.go:117] "RemoveContainer" containerID="c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382"
Sep 15 09:39:20.355198 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:39:20.355178    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382\": container with ID starting with c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382 not found: ID does not exist" containerID="c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382"
Sep 15 09:39:20.355248 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.355208    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382"} err="failed to get container status \"c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382\": rpc error: code = NotFound desc = could not find container \"c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382\": container with ID starting with c146f6cb72901629abc0c2b5dfdb49b10a5d3e24817f1f632ce06510da08f382 not found: ID does not exist"
Sep 15 09:39:20.361363 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.361337    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"]
Sep 15 09:39:20.385591 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.385549    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-08aa53-c9545575b-m56g4"]
Sep 15 09:39:20.878240 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:20.878209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:21.571330 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:21.571299    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:22.055697 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:22.055662    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f770217-796e-4168-aa18-d3fc456249cc" path="/var/lib/kubelet/pods/8f770217-796e-4168-aa18-d3fc456249cc/volumes"
Sep 15 09:39:22.376318 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:22.376282    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:24.911925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:24.911891    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"]
Sep 15 09:39:25.007423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:25.007391    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:39:26.217210 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:26.217171    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:39:39.931702 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:39.931660    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph" podUID="062dfa28-2363-44a6-bc35-b7e683807b95" containerName="main" containerID="cri-o://08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7" gracePeriod=45
Sep 15 09:39:40.178555 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.178528    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:39:40.240884 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.240793    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5jcs\" (UniqueName: \"kubernetes.io/projected/062dfa28-2363-44a6-bc35-b7e683807b95-kube-api-access-w5jcs\") pod \"062dfa28-2363-44a6-bc35-b7e683807b95\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") "
Sep 15 09:39:40.241029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.240887    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-model-cache\") pod \"062dfa28-2363-44a6-bc35-b7e683807b95\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") "
Sep 15 09:39:40.241029 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241003    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-home\") pod \"062dfa28-2363-44a6-bc35-b7e683807b95\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") "
Sep 15 09:39:40.241110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241050    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062dfa28-2363-44a6-bc35-b7e683807b95-tls-certs\") pod \"062dfa28-2363-44a6-bc35-b7e683807b95\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") "
Sep 15 09:39:40.241110 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241088    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-tmp-dir\") pod \"062dfa28-2363-44a6-bc35-b7e683807b95\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") "
Sep 15 09:39:40.241215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241127    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-dshm\") pod \"062dfa28-2363-44a6-bc35-b7e683807b95\" (UID: \"062dfa28-2363-44a6-bc35-b7e683807b95\") "
Sep 15 09:39:40.241273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241208    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-model-cache" (OuterVolumeSpecName: "model-cache") pod "062dfa28-2363-44a6-bc35-b7e683807b95" (UID: "062dfa28-2363-44a6-bc35-b7e683807b95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:39:40.241273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241256    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-home" (OuterVolumeSpecName: "home") pod "062dfa28-2363-44a6-bc35-b7e683807b95" (UID: "062dfa28-2363-44a6-bc35-b7e683807b95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:39:40.241378 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "062dfa28-2363-44a6-bc35-b7e683807b95" (UID: "062dfa28-2363-44a6-bc35-b7e683807b95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:39:40.241616 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241587    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:40.241711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241620    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:40.241711 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.241634    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:40.243249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.243221    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062dfa28-2363-44a6-bc35-b7e683807b95-kube-api-access-w5jcs" (OuterVolumeSpecName: "kube-api-access-w5jcs") pod "062dfa28-2363-44a6-bc35-b7e683807b95" (UID: "062dfa28-2363-44a6-bc35-b7e683807b95"). InnerVolumeSpecName "kube-api-access-w5jcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:39:40.243349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.243257    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062dfa28-2363-44a6-bc35-b7e683807b95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "062dfa28-2363-44a6-bc35-b7e683807b95" (UID: "062dfa28-2363-44a6-bc35-b7e683807b95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:39:40.243349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.243310    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-dshm" (OuterVolumeSpecName: "dshm") pod "062dfa28-2363-44a6-bc35-b7e683807b95" (UID: "062dfa28-2363-44a6-bc35-b7e683807b95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:39:40.342218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.342174    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5jcs\" (UniqueName: \"kubernetes.io/projected/062dfa28-2363-44a6-bc35-b7e683807b95-kube-api-access-w5jcs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:40.342218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.342209    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062dfa28-2363-44a6-bc35-b7e683807b95-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:40.342218 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.342224    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062dfa28-2363-44a6-bc35-b7e683807b95-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:39:40.458034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.457993    2571 generic.go:356] "Generic (PLEG): container finished" podID="062dfa28-2363-44a6-bc35-b7e683807b95" containerID="08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7" exitCode=0
Sep 15 09:39:40.458034 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.458038    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"
Sep 15 09:39:40.458317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.458085    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph" event={"ID":"062dfa28-2363-44a6-bc35-b7e683807b95","Type":"ContainerDied","Data":"08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7"}
Sep 15 09:39:40.458317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.458144    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph" event={"ID":"062dfa28-2363-44a6-bc35-b7e683807b95","Type":"ContainerDied","Data":"4df943600d5ed52495184ec875c97434023005141b7c9a199930b774a1543f1d"}
Sep 15 09:39:40.458317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.458168    2571 scope.go:117] "RemoveContainer" containerID="08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7"
Sep 15 09:39:40.468250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.468231    2571 scope.go:117] "RemoveContainer" containerID="08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7"
Sep 15 09:39:40.468571 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:39:40.468549    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7\": container with ID starting with 08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7 not found: ID does not exist" containerID="08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7"
Sep 15 09:39:40.468648 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.468584    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7"} err="failed to get container status \"08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7\": rpc error: code = NotFound desc = could not find container \"08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7\": container with ID starting with 08d68a70543b0ed8dcc6b48d8fc4b25dff594b50866d73431828b81a94bb70a7 not found: ID does not exist"
Sep 15 09:39:40.481020 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.480992    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"]
Sep 15 09:39:40.484594 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:40.484568    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-08aa53/ipp-route-08aa53-kserve-55686c8887-bclph"]
Sep 15 09:39:42.054771 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:39:42.054737    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="062dfa28-2363-44a6-bc35-b7e683807b95" path="/var/lib/kubelet/pods/062dfa28-2363-44a6-bc35-b7e683807b95/volumes"
Sep 15 09:40:02.117433 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:02.117400    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:02.325610 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:02.325580    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:08.021528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:08.021494    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:08.147545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:08.147512    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:08.270055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:08.270018    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"]
Sep 15 09:40:08.270307 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:08.270279    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy" containerID="cri-o://60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255" gracePeriod=30
Sep 15 09:40:08.294414 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:08.294340    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:08.792561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:08.792522    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:09.679636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:09.679604    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:10.480939 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:10.480903    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:11.170414 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:11.170381    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:11.873572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:11.873541    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:13.526337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.526311    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:40:13.577386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577299    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxdtj\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-kube-api-access-vxdtj\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577374    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-credential-socket\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577419    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-data\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577452    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-envoy\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577619 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577597    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-certs\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577664    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-token\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577663    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:13.577782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577689    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-socket\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577715    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd110e9-fcc1-4830-940c-7c8bbe919831-istiod-ca-cert\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.577782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577717    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:13.577782 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577744    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-podinfo\") pod \"8bd110e9-fcc1-4830-940c-7c8bbe919831\" (UID: \"8bd110e9-fcc1-4830-940c-7c8bbe919831\") "
Sep 15 09:40:13.578192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:13.578192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.577981    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:13.578192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.578160    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd110e9-fcc1-4830-940c-7c8bbe919831-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:40:13.578317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.578225    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.578317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.578247    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.578317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.578261    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.578317 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.578278    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.579758 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.579722    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:13.579758 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.579734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-kube-api-access-vxdtj" (OuterVolumeSpecName: "kube-api-access-vxdtj") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "kube-api-access-vxdtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:13.580147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.580125    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:13.580223 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.580198    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bd110e9-fcc1-4830-940c-7c8bbe919831" (UID: "8bd110e9-fcc1-4830-940c-7c8bbe919831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:40:13.624924 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.624885    2571 generic.go:356] "Generic (PLEG): container finished" podID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerID="60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255" exitCode=0
Sep 15 09:40:13.625080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.624940    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"
Sep 15 09:40:13.625080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.624952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" event={"ID":"8bd110e9-fcc1-4830-940c-7c8bbe919831","Type":"ContainerDied","Data":"60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255"}
Sep 15 09:40:13.625080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.625039    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd" event={"ID":"8bd110e9-fcc1-4830-940c-7c8bbe919831","Type":"ContainerDied","Data":"b4396c11e9d43516c47a0299e96a81b144607aea143e2065a224c0d8771a7c05"}
Sep 15 09:40:13.625080 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.625069    2571 scope.go:117] "RemoveContainer" containerID="60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255"
Sep 15 09:40:13.634265 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.634245    2571 scope.go:117] "RemoveContainer" containerID="710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"
Sep 15 09:40:13.642152 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.642134    2571 scope.go:117] "RemoveContainer" containerID="60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255"
Sep 15 09:40:13.642419 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:40:13.642401    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255\": container with ID starting with 60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255 not found: ID does not exist" containerID="60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255"
Sep 15 09:40:13.642461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.642426    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255"} err="failed to get container status \"60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255\": rpc error: code = NotFound desc = could not find container \"60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255\": container with ID starting with 60eb5cce8d5d02dda662a11ae431cb45ce00952970d0c2aadb76822051907255 not found: ID does not exist"
Sep 15 09:40:13.642461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.642442    2571 scope.go:117] "RemoveContainer" containerID="710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"
Sep 15 09:40:13.642689 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:40:13.642672    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91\": container with ID starting with 710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91 not found: ID does not exist" containerID="710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"
Sep 15 09:40:13.642741 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.642693    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91"} err="failed to get container status \"710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91\": rpc error: code = NotFound desc = could not find container \"710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91\": container with ID starting with 710dd8c1c19c483151d075ccd1ae5e3302e7555bd7dcf3ce7df8176316a23c91 not found: ID does not exist"
Sep 15 09:40:13.646151 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.646125    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"]
Sep 15 09:40:13.650124 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.650098    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-08aa53-openshift-default-76b696fbc4-dm8hd"]
Sep 15 09:40:13.679315 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.679286    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.679315 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.679315    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd110e9-fcc1-4830-940c-7c8bbe919831-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.679474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.679325    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.679474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.679335    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdtj\" (UniqueName: \"kubernetes.io/projected/8bd110e9-fcc1-4830-940c-7c8bbe919831-kube-api-access-vxdtj\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:13.679474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:13.679344    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd110e9-fcc1-4830-940c-7c8bbe919831-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:14.055125 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:14.055090    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" path="/var/lib/kubelet/pods/8bd110e9-fcc1-4830-940c-7c8bbe919831/volumes"
Sep 15 09:40:19.356640 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:19.356611    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:19.429474 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:19.429441    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:19.528802 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:19.528769    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:19.586000 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:19.585966    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:19.964963 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:19.964929    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:20.067877 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:20.067845    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:20.709453 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:20.709416    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:21.580077 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:21.580038    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:21.607891 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:21.607857    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:40:25.777100 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:25.777065    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:25.875469 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:25.875432    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:26.005204 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:26.005165    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"]
Sep 15 09:40:26.005396 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:26.005375    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy" containerID="cri-o://1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111" gracePeriod=30
Sep 15 09:40:26.041440 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:26.041348    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:26.575109 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:26.575049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:27.409099 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:27.409062    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:28.099328 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:28.099296    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:28.873881 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:28.873849    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:31.259160 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.259132    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:40:31.349101 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349058    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-socket\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349135    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-podinfo\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349170    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f44j8\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-kube-api-access-f44j8\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349302 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349224    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-envoy\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349348    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-token\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349377    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:31.349473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349385    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istiod-ca-cert\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349454    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-certs\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349515    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-data\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349682 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349574    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-credential-socket\") pod \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\" (UID: \"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9\") "
Sep 15 09:40:31.349783 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349690    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:40:31.350120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:31.350120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.349994    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.350120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.350013    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.350120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.350019    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:31.350120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.350027    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.350120 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.350081    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:31.351558 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.351526    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:31.351672 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.351641    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:31.351727 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.351680    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:40:31.351798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.351774    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-kube-api-access-f44j8" (OuterVolumeSpecName: "kube-api-access-f44j8") pod "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" (UID: "8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9"). InnerVolumeSpecName "kube-api-access-f44j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:31.450720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.450680    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.450720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.450717    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f44j8\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-kube-api-access-f44j8\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.450985 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.450731    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.450985 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.450743    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.450985 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.450755    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.450985 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.450765    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:31.714379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.714280    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerID="1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111" exitCode=0
Sep 15 09:40:31.714379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.714333    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"
Sep 15 09:40:31.714379 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.714332    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" event={"ID":"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9","Type":"ContainerDied","Data":"1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111"}
Sep 15 09:40:31.714658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.714398    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44" event={"ID":"8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9","Type":"ContainerDied","Data":"bc7c8962757ebb0f5b2c9c752cf76b93188b3f3b1f6aa720544801700c023517"}
Sep 15 09:40:31.714658 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.714411    2571 scope.go:117] "RemoveContainer" containerID="1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111"
Sep 15 09:40:31.723490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.723469    2571 scope.go:117] "RemoveContainer" containerID="881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"
Sep 15 09:40:31.731964 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.731941    2571 scope.go:117] "RemoveContainer" containerID="1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111"
Sep 15 09:40:31.732259 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:40:31.732238    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111\": container with ID starting with 1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111 not found: ID does not exist" containerID="1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111"
Sep 15 09:40:31.732322 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.732268    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111"} err="failed to get container status \"1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111\": rpc error: code = NotFound desc = could not find container \"1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111\": container with ID starting with 1e22ed8cb955f0643f80cebe12712f4be59bf01edc2f3077efde48a7e9b3b111 not found: ID does not exist"
Sep 15 09:40:31.732322 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.732285    2571 scope.go:117] "RemoveContainer" containerID="881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"
Sep 15 09:40:31.732578 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:40:31.732561    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130\": container with ID starting with 881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130 not found: ID does not exist" containerID="881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"
Sep 15 09:40:31.732636 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.732581    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130"} err="failed to get container status \"881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130\": rpc error: code = NotFound desc = could not find container \"881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130\": container with ID starting with 881b40b3d4d8ae7ee8173c3c4efbddebae30ecce6914a103f761e812e402a130 not found: ID does not exist"
Sep 15 09:40:31.736345 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.736319    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"]
Sep 15 09:40:31.741181 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:31.741158    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6920e5-openshift-default-66bff957cd-6pq44"]
Sep 15 09:40:32.055455 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:32.055370    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" path="/var/lib/kubelet/pods/8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9/volumes"
Sep 15 09:40:37.316303 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:37.316266    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:37.399742 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:37.399707    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:37.459052 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:37.459012    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:37.580259 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:37.580175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:38.369333 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:38.369291    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:39.809093 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:39.809054    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:40:43.172737 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:43.172695    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:43.238988 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:43.238934    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:43.425461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:43.425355    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"]
Sep 15 09:40:43.425641 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:43.425619    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy" containerID="cri-o://161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6" gracePeriod=30
Sep 15 09:40:43.440600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:43.440558    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:44.685914 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:44.685879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:45.381033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:45.380991    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:46.509299 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:46.509265    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:48.680663 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.680634    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:40:48.715349 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715313    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-certs\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715362    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istiod-ca-cert\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715416    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-credential-socket\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715437    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-795tj\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-kube-api-access-795tj\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715547 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715463    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-socket\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715598    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-envoy\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715628    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:48.715791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715676    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-data\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715718    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-token\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.715791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715766    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:48.715791 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715779    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:48.716137 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715784    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-podinfo\") pod \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\" (UID: \"2f0881a8-e2ba-42f7-bdea-20ec581d6bef\") "
Sep 15 09:40:48.716137 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.715917    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:40:48.716137 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.716023    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:48.716305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.716176    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.716305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.716195    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.716305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.716210    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.716305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.716222    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.716305 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.716230    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.718017 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.717987    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:48.718131 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.718038    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:40:48.718235 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.718219    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-kube-api-access-795tj" (OuterVolumeSpecName: "kube-api-access-795tj") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "kube-api-access-795tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:40:48.718295 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.718225    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f0881a8-e2ba-42f7-bdea-20ec581d6bef" (UID: "2f0881a8-e2ba-42f7-bdea-20ec581d6bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:40:48.797369 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.797337    2571 generic.go:356] "Generic (PLEG): container finished" podID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerID="161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6" exitCode=0
Sep 15 09:40:48.797536 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.797386    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"
Sep 15 09:40:48.797536 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.797423    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" event={"ID":"2f0881a8-e2ba-42f7-bdea-20ec581d6bef","Type":"ContainerDied","Data":"161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6"}
Sep 15 09:40:48.797536 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.797476    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79" event={"ID":"2f0881a8-e2ba-42f7-bdea-20ec581d6bef","Type":"ContainerDied","Data":"54eabb7e2318f33e08f8103b2f0b4098b8ca31f400f52a495081ccec02825874"}
Sep 15 09:40:48.797536 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.797499    2571 scope.go:117] "RemoveContainer" containerID="161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6"
Sep 15 09:40:48.809741 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.809719    2571 scope.go:117] "RemoveContainer" containerID="3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"
Sep 15 09:40:48.817568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.817543    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795tj\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-kube-api-access-795tj\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.817568 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.817566    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.817717 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.817577    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.817717 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.817585    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f0881a8-e2ba-42f7-bdea-20ec581d6bef-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:40:48.818169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.818152    2571 scope.go:117] "RemoveContainer" containerID="161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6"
Sep 15 09:40:48.818447 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:40:48.818432    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6\": container with ID starting with 161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6 not found: ID does not exist" containerID="161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6"
Sep 15 09:40:48.818498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.818455    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6"} err="failed to get container status \"161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6\": rpc error: code = NotFound desc = could not find container \"161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6\": container with ID starting with 161401d15d99c863df120fe882cab72d6540db53d0090bcf2a72c3bfd4841fd6 not found: ID does not exist"
Sep 15 09:40:48.818498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.818472    2571 scope.go:117] "RemoveContainer" containerID="3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"
Sep 15 09:40:48.818734 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:40:48.818718    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b\": container with ID starting with 3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b not found: ID does not exist" containerID="3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"
Sep 15 09:40:48.818794 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.818735    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b"} err="failed to get container status \"3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b\": rpc error: code = NotFound desc = could not find container \"3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b\": container with ID starting with 3586f12988cd9e3e3162ff43f03e8fb57ec324efa6b9eb174b85f742a42db35b not found: ID does not exist"
Sep 15 09:40:48.823668 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.823638    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"]
Sep 15 09:40:48.830902 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:48.830871    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f79800-openshift-default-7b749854cb-xhn79"]
Sep 15 09:40:50.055416 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:50.055384    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" path="/var/lib/kubelet/pods/2f0881a8-e2ba-42f7-bdea-20ec581d6bef/volumes"
Sep 15 09:40:54.301163 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:54.301072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:54.371498 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:54.371459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:54.502105 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:54.502067    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:54.600798 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:54.600768    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:54.861513 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:54.861425    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:54.987729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:54.987696    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:40:57.109135 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:40:57.109096    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:41:00.808551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:00.808507    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:41:00.886054 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:00.886013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:41:01.051649 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:01.051616    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"]
Sep 15 09:41:01.051904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:01.051879    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy" containerID="cri-o://fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27" gracePeriod=30
Sep 15 09:41:01.073556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:01.073471    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:41:05.008644 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:05.008609    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:41:06.319654 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.319574    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:41:06.399855 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.399790    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-envoy\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.399855 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.399855    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-istio-token\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.399883    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-credential-socket\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.399948    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f41605-709c-4a82-96f3-8b285971b171-istio-podinfo\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.399991    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-certs\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400014    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-socket\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400038    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9rb4\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-kube-api-access-g9rb4\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400064    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-data\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400123 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400115    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f41605-709c-4a82-96f3-8b285971b171-istiod-ca-cert\") pod \"32f41605-709c-4a82-96f3-8b285971b171\" (UID: \"32f41605-709c-4a82-96f3-8b285971b171\") "
Sep 15 09:41:06.400493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400299    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:06.400493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400389    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:06.400493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400483    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.400665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400504    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-workload-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.400665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400531    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:06.400665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400565    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-data" (OuterVolumeSpecName: "istio-data") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:06.400665 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.400610    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f41605-709c-4a82-96f3-8b285971b171-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:41:06.402249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.402218    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:41:06.402360 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.402256    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-istio-token" (OuterVolumeSpecName: "istio-token") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:06.402360 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.402270    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f41605-709c-4a82-96f3-8b285971b171-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 09:41:06.402458 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.402441    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-kube-api-access-g9rb4" (OuterVolumeSpecName: "kube-api-access-g9rb4") pod "32f41605-709c-4a82-96f3-8b285971b171" (UID: "32f41605-709c-4a82-96f3-8b285971b171"). InnerVolumeSpecName "kube-api-access-g9rb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:41:06.501350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501309    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-envoy\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.501350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501341    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-istio-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.501350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501349    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-credential-socket\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.501350 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501360    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f41605-709c-4a82-96f3-8b285971b171-istio-podinfo\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.501623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501369    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rb4\" (UniqueName: \"kubernetes.io/projected/32f41605-709c-4a82-96f3-8b285971b171-kube-api-access-g9rb4\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.501623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501378    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f41605-709c-4a82-96f3-8b285971b171-istio-data\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.501623 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.501386    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f41605-709c-4a82-96f3-8b285971b171-istiod-ca-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:41:06.884709 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.884673    2571 generic.go:356] "Generic (PLEG): container finished" podID="32f41605-709c-4a82-96f3-8b285971b171" containerID="fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27" exitCode=0
Sep 15 09:41:06.884904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.884716    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"
Sep 15 09:41:06.884904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.884753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" event={"ID":"32f41605-709c-4a82-96f3-8b285971b171","Type":"ContainerDied","Data":"fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27"}
Sep 15 09:41:06.884904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.884802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp" event={"ID":"32f41605-709c-4a82-96f3-8b285971b171","Type":"ContainerDied","Data":"b1dd2831000acfdcc48da7526b367784af2a634bd1477b91c90b8180aa51f4fa"}
Sep 15 09:41:06.884904 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.884841    2571 scope.go:117] "RemoveContainer" containerID="fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27"
Sep 15 09:41:06.894327 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.894306    2571 scope.go:117] "RemoveContainer" containerID="3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"
Sep 15 09:41:06.902454 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.902429    2571 scope.go:117] "RemoveContainer" containerID="fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27"
Sep 15 09:41:06.902759 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:41:06.902739    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27\": container with ID starting with fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27 not found: ID does not exist" containerID="fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27"
Sep 15 09:41:06.902808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.902770    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27"} err="failed to get container status \"fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27\": rpc error: code = NotFound desc = could not find container \"fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27\": container with ID starting with fad54112767ab390a529c388cf88ee62e6480700919e46323c8d6031e5a9ad27 not found: ID does not exist"
Sep 15 09:41:06.902808 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.902791    2571 scope.go:117] "RemoveContainer" containerID="3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"
Sep 15 09:41:06.903075 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:41:06.903054    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619\": container with ID starting with 3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619 not found: ID does not exist" containerID="3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"
Sep 15 09:41:06.903114 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.903080    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619"} err="failed to get container status \"3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619\": rpc error: code = NotFound desc = could not find container \"3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619\": container with ID starting with 3b5caa072d0dfd5bda22907eebece989be84717b6f972c71e271378c467c0619 not found: ID does not exist"
Sep 15 09:41:06.906804 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.906775    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"]
Sep 15 09:41:06.910164 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:06.910141    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8dd3bd-openshift-default-68786d948-7xwtp"]
Sep 15 09:41:08.054719 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:08.054684    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f41605-709c-4a82-96f3-8b285971b171" path="/var/lib/kubelet/pods/32f41605-709c-4a82-96f3-8b285971b171/volumes"
Sep 15 09:41:48.145762 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:48.145725    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:41:48.153441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:41:48.153417    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:42:25.240584 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:42:25.240548    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:42:25.716015 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:42:25.715980    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:42:27.130589 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:42:27.130556    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:42:36.437386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:42:36.437349    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:42:36.493304 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:42:36.493271    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:42:52.104979 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:42:52.104891    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:43:00.207729 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:43:00.207688    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:43:05.701170 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:43:05.701136    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:43:07.189850 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:43:07.189793    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:43:08.005842 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:43:08.005797    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:43:22.802537 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:43:22.802497    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:43:32.809760 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:43:32.809720    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:44:04.175454 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:44:04.175417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:44:04.225374 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:44:04.225346    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:44:20.274493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:44:20.274458    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:45:00.601248 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:00.601211    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:45:04.328263 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:04.328201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:45:08.808180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:08.808144    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:45:15.002556 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:15.002520    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:45:33.605925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.605886    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"]
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606232    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062dfa28-2363-44a6-bc35-b7e683807b95" containerName="main"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606243    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="062dfa28-2363-44a6-bc35-b7e683807b95" containerName="main"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606254    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606261    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606270    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f770217-796e-4168-aa18-d3fc456249cc" containerName="maas-api"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606276    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f770217-796e-4168-aa18-d3fc456249cc" containerName="maas-api"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606283    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606288    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606298    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606303    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606317    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606322    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606330    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606335    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606341    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy"
Sep 15 09:45:33.606408 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606345    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606418    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606430    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606437    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606444    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606453    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="062dfa28-2363-44a6-bc35-b7e683807b95" containerName="main"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606459    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606465    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f770217-796e-4168-aa18-d3fc456249cc" containerName="maas-api"
Sep 15 09:45:33.606919 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.606470    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f41605-709c-4a82-96f3-8b285971b171" containerName="istio-proxy"
Sep 15 09:45:33.609398 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.609380    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.612023 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.611999    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-34863e4a-kserve-self-signed-certs\""
Sep 15 09:45:33.617758 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.617732    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"]
Sep 15 09:45:33.703917 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.703883    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:45:33.711405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.711376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-dshm\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.711561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.711424    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-home\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.711561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.711445    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tls-certs\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.711561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.711466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7zv\" (UniqueName: \"kubernetes.io/projected/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-kube-api-access-db7zv\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.711561 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.711534    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tmp-dir\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.711720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.711575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-model-cache\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.812545 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tmp-dir\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.812726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-model-cache\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.812726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-dshm\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.812726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812648    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-home\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.812726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812679    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tls-certs\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.812726 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7zv\" (UniqueName: \"kubernetes.io/projected/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-kube-api-access-db7zv\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.813027 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.812981    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tmp-dir\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.813092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.813029    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-model-cache\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.813092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.813074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-home\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.814944 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.814917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-dshm\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.815243 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.815223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tls-certs\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.820441 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.820416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7zv\" (UniqueName: \"kubernetes.io/projected/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-kube-api-access-db7zv\") pod \"e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") " pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:33.919447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:33.919361    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:34.191786 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:34.191758    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"]
Sep 15 09:45:34.194392 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:45:34.194362    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0085e62f_ee22_4a96_8131_3b1c00fa4e9b.slice/crio-97ea90d12f35609e3e672d27d00677effe755b8f76af2e97f9afbc01b9963a6a WatchSource:0}: Error finding container 97ea90d12f35609e3e672d27d00677effe755b8f76af2e97f9afbc01b9963a6a: Status 404 returned error can't find the container with id 97ea90d12f35609e3e672d27d00677effe755b8f76af2e97f9afbc01b9963a6a
Sep 15 09:45:35.158505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:35.158463    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s" event={"ID":"0085e62f-ee22-4a96-8131-3b1c00fa4e9b","Type":"ContainerStarted","Data":"e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913"}
Sep 15 09:45:35.158929 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:35.158515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s" event={"ID":"0085e62f-ee22-4a96-8131-3b1c00fa4e9b","Type":"ContainerStarted","Data":"97ea90d12f35609e3e672d27d00677effe755b8f76af2e97f9afbc01b9963a6a"}
Sep 15 09:45:35.174249 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:35.174201    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s" podStartSLOduration=2.174183172 podStartE2EDuration="2.174183172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:45:35.173031043 +0000 UTC m=+2327.632320608" watchObservedRunningTime="2026-09-15 09:45:35.174183172 +0000 UTC m=+2327.633472713"
Sep 15 09:45:36.408391 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:36.408352    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:45:43.112290 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112190    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"]
Sep 15 09:45:43.112704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112564    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy"
Sep 15 09:45:43.112704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112576    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy"
Sep 15 09:45:43.112704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112594    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy"
Sep 15 09:45:43.112704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112600    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0881a8-e2ba-42f7-bdea-20ec581d6bef" containerName="istio-proxy"
Sep 15 09:45:43.112704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112656    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd110e9-fcc1-4830-940c-7c8bbe919831" containerName="istio-proxy"
Sep 15 09:45:43.112704 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.112664    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1010dd-f739-4cb4-a0d6-38b0ffd8cbe9" containerName="istio-proxy"
Sep 15 09:45:43.115835 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.115809    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.118192 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.118172    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-34863e4a-kserve-self-signed-certs\""
Sep 15 09:45:43.127461 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.127432    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"]
Sep 15 09:45:43.197589 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.197537    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:45:43.201809 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.201781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16172167-5547-4e63-945f-7278a421bb3c-tls-certs\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.201953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.201822    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-home\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.201953 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.201917    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9lmt\" (UniqueName: \"kubernetes.io/projected/16172167-5547-4e63-945f-7278a421bb3c-kube-api-access-z9lmt\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.202047 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.201988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-tmp-dir\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.202047 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.202005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-dshm\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.202154 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.202078    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-model-cache\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.302592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16172167-5547-4e63-945f-7278a421bb3c-tls-certs\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.302592 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-home\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.302905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302726    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9lmt\" (UniqueName: \"kubernetes.io/projected/16172167-5547-4e63-945f-7278a421bb3c-kube-api-access-z9lmt\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.302905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-tmp-dir\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.302905 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-dshm\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.303071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-model-cache\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.303071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.302970    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-home\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.303296 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.303185    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-tmp-dir\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.303296 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.303289    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-model-cache\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.305054 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.305029    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-dshm\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.305236 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.305216    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16172167-5547-4e63-945f-7278a421bb3c-tls-certs\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.310785 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.310754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9lmt\" (UniqueName: \"kubernetes.io/projected/16172167-5547-4e63-945f-7278a421bb3c-kube-api-access-z9lmt\") pod \"e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") " pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.426028 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.425932    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:43.699657 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.699629    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"]
Sep 15 09:45:43.701708 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:45:43.701680    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16172167_5547_4e63_945f_7278a421bb3c.slice/crio-f092808e3cbd4e7a26d909bb757e7c8947a67b0115392da502e4e40d1f84ec0f WatchSource:0}: Error finding container f092808e3cbd4e7a26d909bb757e7c8947a67b0115392da502e4e40d1f84ec0f: Status 404 returned error can't find the container with id f092808e3cbd4e7a26d909bb757e7c8947a67b0115392da502e4e40d1f84ec0f
Sep 15 09:45:43.920014 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.919962    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:43.920195 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.920029    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:43.932644 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:43.932613    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:44.204174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:44.204132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74" event={"ID":"16172167-5547-4e63-945f-7278a421bb3c","Type":"ContainerStarted","Data":"183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b"}
Sep 15 09:45:44.204627 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:44.204181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74" event={"ID":"16172167-5547-4e63-945f-7278a421bb3c","Type":"ContainerStarted","Data":"f092808e3cbd4e7a26d909bb757e7c8947a67b0115392da502e4e40d1f84ec0f"}
Sep 15 09:45:44.216180 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:44.216152    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:45:44.226440 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:44.226367    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74" podStartSLOduration=1.226346714 podStartE2EDuration="1.226346714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:45:44.221176419 +0000 UTC m=+2336.680465960" watchObservedRunningTime="2026-09-15 09:45:44.226346714 +0000 UTC m=+2336.685636264"
Sep 15 09:45:47.203838 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:47.203791    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:45:53.426546 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:53.426512    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:53.426985 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:53.426567    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:53.439298 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:53.439266    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:45:54.262899 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:45:54.262866    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:46:07.579260 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:07.579223    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"]
Sep 15 09:46:07.600237 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:07.600209    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:46:09.406858 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:09.406802    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"]
Sep 15 09:46:09.429229 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:09.429190    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:46:09.902247 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:09.902207    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:11.339224 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:11.339192    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:46:17.304473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:17.304436    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:22.599551 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.599507    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s" podUID="0085e62f-ee22-4a96-8131-3b1c00fa4e9b" containerName="main" containerID="cri-o://e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913" gracePeriod=45
Sep 15 09:46:22.859790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.859716    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:46:22.880077 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880043    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db7zv\" (UniqueName: \"kubernetes.io/projected/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-kube-api-access-db7zv\") pod \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") "
Sep 15 09:46:22.880273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880158    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tls-certs\") pod \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") "
Sep 15 09:46:22.880273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880204    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tmp-dir\") pod \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") "
Sep 15 09:46:22.880273 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880235    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-home\") pod \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") "
Sep 15 09:46:22.880447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880281    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-dshm\") pod \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") "
Sep 15 09:46:22.880447 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880308    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-model-cache\") pod \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\" (UID: \"0085e62f-ee22-4a96-8131-3b1c00fa4e9b\") "
Sep 15 09:46:22.880572 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880514    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0085e62f-ee22-4a96-8131-3b1c00fa4e9b" (UID: "0085e62f-ee22-4a96-8131-3b1c00fa4e9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:22.880926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880676    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:22.880926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880695    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-home" (OuterVolumeSpecName: "home") pod "0085e62f-ee22-4a96-8131-3b1c00fa4e9b" (UID: "0085e62f-ee22-4a96-8131-3b1c00fa4e9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:22.880926 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.880914    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0085e62f-ee22-4a96-8131-3b1c00fa4e9b" (UID: "0085e62f-ee22-4a96-8131-3b1c00fa4e9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:22.882720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.882689    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0085e62f-ee22-4a96-8131-3b1c00fa4e9b" (UID: "0085e62f-ee22-4a96-8131-3b1c00fa4e9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:46:22.882720 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.882709    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-kube-api-access-db7zv" (OuterVolumeSpecName: "kube-api-access-db7zv") pod "0085e62f-ee22-4a96-8131-3b1c00fa4e9b" (UID: "0085e62f-ee22-4a96-8131-3b1c00fa4e9b"). InnerVolumeSpecName "kube-api-access-db7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:46:22.883178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.883146    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-dshm" (OuterVolumeSpecName: "dshm") pod "0085e62f-ee22-4a96-8131-3b1c00fa4e9b" (UID: "0085e62f-ee22-4a96-8131-3b1c00fa4e9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:22.981780 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.981738    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:22.981780 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.981771    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:22.981780 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.981782    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db7zv\" (UniqueName: \"kubernetes.io/projected/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-kube-api-access-db7zv\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:22.981780 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.981790    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:22.982178 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:22.981801    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0085e62f-ee22-4a96-8131-3b1c00fa4e9b-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:23.401165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.401067    2571 generic.go:356] "Generic (PLEG): container finished" podID="0085e62f-ee22-4a96-8131-3b1c00fa4e9b" containerID="e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913" exitCode=0
Sep 15 09:46:23.401165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.401109    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s" event={"ID":"0085e62f-ee22-4a96-8131-3b1c00fa4e9b","Type":"ContainerDied","Data":"e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913"}
Sep 15 09:46:23.401165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.401164    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s" event={"ID":"0085e62f-ee22-4a96-8131-3b1c00fa4e9b","Type":"ContainerDied","Data":"97ea90d12f35609e3e672d27d00677effe755b8f76af2e97f9afbc01b9963a6a"}
Sep 15 09:46:23.401486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.401187    2571 scope.go:117] "RemoveContainer" containerID="e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913"
Sep 15 09:46:23.401486 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.401126    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"
Sep 15 09:46:23.411250 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.411231    2571 scope.go:117] "RemoveContainer" containerID="e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913"
Sep 15 09:46:23.411562 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:46:23.411545    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913\": container with ID starting with e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913 not found: ID does not exist" containerID="e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913"
Sep 15 09:46:23.411606 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.411570    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913"} err="failed to get container status \"e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913\": rpc error: code = NotFound desc = could not find container \"e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913\": container with ID starting with e81befa1b3ca17b125da0d2c799ecbe08bd57dc85bed8e1f5b86f693ff549913 not found: ID does not exist"
Sep 15 09:46:23.423751 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.423721    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"]
Sep 15 09:46:23.429175 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:23.429146    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-34863e4a-kserve-696685b6c9-dlv7s"]
Sep 15 09:46:24.054895 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.054849    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0085e62f-ee22-4a96-8131-3b1c00fa4e9b" path="/var/lib/kubelet/pods/0085e62f-ee22-4a96-8131-3b1c00fa4e9b/volumes"
Sep 15 09:46:24.428072 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.428032    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74" podUID="16172167-5547-4e63-945f-7278a421bb3c" containerName="main" containerID="cri-o://183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b" gracePeriod=45
Sep 15 09:46:24.680193 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.680134    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:46:24.697929 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.697892    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-home\") pod \"16172167-5547-4e63-945f-7278a421bb3c\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") "
Sep 15 09:46:24.698088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.697954    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-dshm\") pod \"16172167-5547-4e63-945f-7278a421bb3c\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") "
Sep 15 09:46:24.698088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698006    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16172167-5547-4e63-945f-7278a421bb3c-tls-certs\") pod \"16172167-5547-4e63-945f-7278a421bb3c\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") "
Sep 15 09:46:24.698088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698064    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-tmp-dir\") pod \"16172167-5547-4e63-945f-7278a421bb3c\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") "
Sep 15 09:46:24.698088 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698083    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-model-cache\") pod \"16172167-5547-4e63-945f-7278a421bb3c\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") "
Sep 15 09:46:24.698297 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698106    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9lmt\" (UniqueName: \"kubernetes.io/projected/16172167-5547-4e63-945f-7278a421bb3c-kube-api-access-z9lmt\") pod \"16172167-5547-4e63-945f-7278a421bb3c\" (UID: \"16172167-5547-4e63-945f-7278a421bb3c\") "
Sep 15 09:46:24.698297 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698171    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-home" (OuterVolumeSpecName: "home") pod "16172167-5547-4e63-945f-7278a421bb3c" (UID: "16172167-5547-4e63-945f-7278a421bb3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:24.698407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698299    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16172167-5547-4e63-945f-7278a421bb3c" (UID: "16172167-5547-4e63-945f-7278a421bb3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:24.698407 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698379    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "16172167-5547-4e63-945f-7278a421bb3c" (UID: "16172167-5547-4e63-945f-7278a421bb3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:24.698505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698413    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-home\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:24.698505 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.698430    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-tmp-dir\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:24.700405 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.700373    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-dshm" (OuterVolumeSpecName: "dshm") pod "16172167-5547-4e63-945f-7278a421bb3c" (UID: "16172167-5547-4e63-945f-7278a421bb3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 09:46:24.700501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.700410    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16172167-5547-4e63-945f-7278a421bb3c-kube-api-access-z9lmt" (OuterVolumeSpecName: "kube-api-access-z9lmt") pod "16172167-5547-4e63-945f-7278a421bb3c" (UID: "16172167-5547-4e63-945f-7278a421bb3c"). InnerVolumeSpecName "kube-api-access-z9lmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:46:24.700501 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.700432    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16172167-5547-4e63-945f-7278a421bb3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16172167-5547-4e63-945f-7278a421bb3c" (UID: "16172167-5547-4e63-945f-7278a421bb3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:46:24.799319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.799283    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16172167-5547-4e63-945f-7278a421bb3c-tls-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:24.799319 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.799317    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-model-cache\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:24.799528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.799328    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9lmt\" (UniqueName: \"kubernetes.io/projected/16172167-5547-4e63-945f-7278a421bb3c-kube-api-access-z9lmt\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:24.799528 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:24.799338    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16172167-5547-4e63-945f-7278a421bb3c-dshm\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:25.248491 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.248454    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f4884bd-pqwng"]
Sep 15 09:46:25.249055 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.249031    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0085e62f-ee22-4a96-8131-3b1c00fa4e9b" containerName="main"
Sep 15 09:46:25.249128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.249058    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0085e62f-ee22-4a96-8131-3b1c00fa4e9b" containerName="main"
Sep 15 09:46:25.249128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.249083    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16172167-5547-4e63-945f-7278a421bb3c" containerName="main"
Sep 15 09:46:25.249128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.249094    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="16172167-5547-4e63-945f-7278a421bb3c" containerName="main"
Sep 15 09:46:25.249315 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.249188    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0085e62f-ee22-4a96-8131-3b1c00fa4e9b" containerName="main"
Sep 15 09:46:25.249315 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.249205    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="16172167-5547-4e63-945f-7278a421bb3c" containerName="main"
Sep 15 09:46:25.252502 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.252481    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.256200 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.256172    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ptrgq\""
Sep 15 09:46:25.256337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.256221    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 09:46:25.256337 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.256252    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 09:46:25.256459 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.256443    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 09:46:25.256629 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.256559    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 09:46:25.264457 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.264366    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f4884bd-pqwng"]
Sep 15 09:46:25.266989 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.266799    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"]
Sep 15 09:46:25.270664 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.270640    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.278675 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.278639    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"]
Sep 15 09:46:25.304162 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc3e241-2107-465a-a548-ed0b6ad5ece4-maas-api-tls\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.304351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vm46\" (UniqueName: \"kubernetes.io/projected/1db095bd-035a-4d2b-b2e5-1d71ce848b97-kube-api-access-6vm46\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.304351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-metrics-certs\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.304351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db095bd-035a-4d2b-b2e5-1d71ce848b97-openshift-service-ca\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.304351 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3e241-2107-465a-a548-ed0b6ad5ece4-openshift-service-ca\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.304527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98zg\" (UniqueName: \"kubernetes.io/projected/ffc3e241-2107-465a-a548-ed0b6ad5ece4-kube-api-access-l98zg\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.304527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304424    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-maas-api-tls\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.304527 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.304463    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3e241-2107-465a-a548-ed0b6ad5ece4-metrics-certs\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.405600 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3e241-2107-465a-a548-ed0b6ad5ece4-openshift-service-ca\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.405787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405636    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98zg\" (UniqueName: \"kubernetes.io/projected/ffc3e241-2107-465a-a548-ed0b6ad5ece4-kube-api-access-l98zg\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.405787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-maas-api-tls\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.405787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3e241-2107-465a-a548-ed0b6ad5ece4-metrics-certs\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.405787 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc3e241-2107-465a-a548-ed0b6ad5ece4-maas-api-tls\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.406626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vm46\" (UniqueName: \"kubernetes.io/projected/1db095bd-035a-4d2b-b2e5-1d71ce848b97-kube-api-access-6vm46\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.406626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-metrics-certs\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.406626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.405887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db095bd-035a-4d2b-b2e5-1d71ce848b97-openshift-service-ca\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.406626 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.406507    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3e241-2107-465a-a548-ed0b6ad5ece4-openshift-service-ca\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.407169 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.407147    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db095bd-035a-4d2b-b2e5-1d71ce848b97-openshift-service-ca\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.408389 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.408365    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:25.409038 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.409015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-maas-api-tls\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.409128 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.409043    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffc3e241-2107-465a-a548-ed0b6ad5ece4-maas-api-tls\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.409204 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.409188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffc3e241-2107-465a-a548-ed0b6ad5ece4-metrics-certs\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.409266 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.409251    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-metrics-certs\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.413199 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413175    2571 generic.go:356] "Generic (PLEG): container finished" podID="16172167-5547-4e63-945f-7278a421bb3c" containerID="183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b" exitCode=0
Sep 15 09:46:25.413308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413223    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"
Sep 15 09:46:25.413308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413241    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74" event={"ID":"16172167-5547-4e63-945f-7278a421bb3c","Type":"ContainerDied","Data":"183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b"}
Sep 15 09:46:25.413308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74" event={"ID":"16172167-5547-4e63-945f-7278a421bb3c","Type":"ContainerDied","Data":"f092808e3cbd4e7a26d909bb757e7c8947a67b0115392da502e4e40d1f84ec0f"}
Sep 15 09:46:25.413308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98zg\" (UniqueName: \"kubernetes.io/projected/ffc3e241-2107-465a-a548-ed0b6ad5ece4-kube-api-access-l98zg\") pod \"maas-api-94fbc46d8-4nlg5\" (UID: \"ffc3e241-2107-465a-a548-ed0b6ad5ece4\") " pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.413308 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413289    2571 scope.go:117] "RemoveContainer" containerID="183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b"
Sep 15 09:46:25.413560 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.413353    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vm46\" (UniqueName: \"kubernetes.io/projected/1db095bd-035a-4d2b-b2e5-1d71ce848b97-kube-api-access-6vm46\") pod \"maas-api-f4884bd-pqwng\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") " pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.421921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.421903    2571 scope.go:117] "RemoveContainer" containerID="183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b"
Sep 15 09:46:25.422212 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:46:25.422194    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b\": container with ID starting with 183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b not found: ID does not exist" containerID="183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b"
Sep 15 09:46:25.422276 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.422219    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b"} err="failed to get container status \"183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b\": rpc error: code = NotFound desc = could not find container \"183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b\": container with ID starting with 183e87f564b0b3ce3c4f59a2743b06fbff06f2eb84f40110fa995a7ff4e25d1b not found: ID does not exist"
Sep 15 09:46:25.433508 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.433478    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"]
Sep 15 09:46:25.437030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.437006    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-34863e4a-kserve-b978ff9d7-hdt74"]
Sep 15 09:46:25.571322 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.571226    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:25.584709 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.584681    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:25.852480 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.852448    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f4884bd-pqwng"]
Sep 15 09:46:25.856325 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:46:25.856292    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db095bd_035a_4d2b_b2e5_1d71ce848b97.slice/crio-db8cc9952c596e93a3657b78f6cbf1b237f6cd962bcf9f695189e14156f98ca0 WatchSource:0}: Error finding container db8cc9952c596e93a3657b78f6cbf1b237f6cd962bcf9f695189e14156f98ca0: Status 404 returned error can't find the container with id db8cc9952c596e93a3657b78f6cbf1b237f6cd962bcf9f695189e14156f98ca0
Sep 15 09:46:25.857790 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.857765    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 09:46:25.876346 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:25.876318    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"]
Sep 15 09:46:25.879221 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:46:25.879197    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc3e241_2107_465a_a548_ed0b6ad5ece4.slice/crio-eb4c92253020c53a35d44e8a1f52ba5c4235c11aebb62d835be88067b0ecf6d3 WatchSource:0}: Error finding container eb4c92253020c53a35d44e8a1f52ba5c4235c11aebb62d835be88067b0ecf6d3: Status 404 returned error can't find the container with id eb4c92253020c53a35d44e8a1f52ba5c4235c11aebb62d835be88067b0ecf6d3
Sep 15 09:46:26.057688 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:26.057644    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16172167-5547-4e63-945f-7278a421bb3c" path="/var/lib/kubelet/pods/16172167-5547-4e63-945f-7278a421bb3c/volumes"
Sep 15 09:46:26.419667 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:26.419629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5" event={"ID":"ffc3e241-2107-465a-a548-ed0b6ad5ece4","Type":"ContainerStarted","Data":"4d108ad035369d8c58fa23355bf72dc7dfd76b1453b7ecad39e5b3da6298591b"}
Sep 15 09:46:26.420136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:26.419684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5" event={"ID":"ffc3e241-2107-465a-a548-ed0b6ad5ece4","Type":"ContainerStarted","Data":"eb4c92253020c53a35d44e8a1f52ba5c4235c11aebb62d835be88067b0ecf6d3"}
Sep 15 09:46:26.420136 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:26.419735    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:26.421955 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:26.421930    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng" event={"ID":"1db095bd-035a-4d2b-b2e5-1d71ce848b97","Type":"ContainerStarted","Data":"db8cc9952c596e93a3657b78f6cbf1b237f6cd962bcf9f695189e14156f98ca0"}
Sep 15 09:46:26.435165 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:26.435116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5" podStartSLOduration=1.024408485 podStartE2EDuration="1.435100926s" totalImagesPullingTime="410.692441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:46:26.434851086 +0000 UTC m=+2378.894140628" watchObservedRunningTime="2026-09-15 09:46:26.435100926 +0000 UTC m=+2378.894390466"
Sep 15 09:46:27.428809 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:27.428775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng" event={"ID":"1db095bd-035a-4d2b-b2e5-1d71ce848b97","Type":"ContainerStarted","Data":"6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb"}
Sep 15 09:46:27.429251 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:27.429032    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:27.445736 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:27.445683    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng" podStartSLOduration=1.672258395 podStartE2EDuration="2.445668082s" totalImagesPullingTime="773.409687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 09:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:46:27.445595512 +0000 UTC m=+2379.904885054" watchObservedRunningTime="2026-09-15 09:46:27.445668082 +0000 UTC m=+2379.904957626"
Sep 15 09:46:32.435012 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.434982    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-94fbc46d8-4nlg5"
Sep 15 09:46:32.481085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.481049    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f4884bd-pqwng"]
Sep 15 09:46:32.481300 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.481272    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng" podUID="1db095bd-035a-4d2b-b2e5-1d71ce848b97" containerName="maas-api" containerID="cri-o://6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb" gracePeriod=30
Sep 15 09:46:32.487108 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.487082    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:32.627092 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.627057    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:32.734271 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.734243    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:32.778878 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.778843    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-metrics-certs\") pod \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") "
Sep 15 09:46:32.779058 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.778925    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-maas-api-tls\") pod \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") "
Sep 15 09:46:32.779058 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.778975    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db095bd-035a-4d2b-b2e5-1d71ce848b97-openshift-service-ca\") pod \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") "
Sep 15 09:46:32.779058 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.779014    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vm46\" (UniqueName: \"kubernetes.io/projected/1db095bd-035a-4d2b-b2e5-1d71ce848b97-kube-api-access-6vm46\") pod \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\" (UID: \"1db095bd-035a-4d2b-b2e5-1d71ce848b97\") "
Sep 15 09:46:32.779437 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.779407    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db095bd-035a-4d2b-b2e5-1d71ce848b97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1db095bd-035a-4d2b-b2e5-1d71ce848b97" (UID: "1db095bd-035a-4d2b-b2e5-1d71ce848b97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 09:46:32.781147 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.781119    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1db095bd-035a-4d2b-b2e5-1d71ce848b97" (UID: "1db095bd-035a-4d2b-b2e5-1d71ce848b97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:46:32.781235 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.781186    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db095bd-035a-4d2b-b2e5-1d71ce848b97-kube-api-access-6vm46" (OuterVolumeSpecName: "kube-api-access-6vm46") pod "1db095bd-035a-4d2b-b2e5-1d71ce848b97" (UID: "1db095bd-035a-4d2b-b2e5-1d71ce848b97"). InnerVolumeSpecName "kube-api-access-6vm46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 09:46:32.781372 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.781354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1db095bd-035a-4d2b-b2e5-1d71ce848b97" (UID: "1db095bd-035a-4d2b-b2e5-1d71ce848b97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 09:46:32.880806 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.880771    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-maas-api-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:32.880806 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.880802    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db095bd-035a-4d2b-b2e5-1d71ce848b97-openshift-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:32.880806 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.880813    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vm46\" (UniqueName: \"kubernetes.io/projected/1db095bd-035a-4d2b-b2e5-1d71ce848b97-kube-api-access-6vm46\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:32.881059 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:32.880821    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db095bd-035a-4d2b-b2e5-1d71ce848b97-metrics-certs\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Sep 15 09:46:33.459215 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.459172    2571 generic.go:356] "Generic (PLEG): container finished" podID="1db095bd-035a-4d2b-b2e5-1d71ce848b97" containerID="6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb" exitCode=0
Sep 15 09:46:33.459615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.459221    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng"
Sep 15 09:46:33.459615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.459252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng" event={"ID":"1db095bd-035a-4d2b-b2e5-1d71ce848b97","Type":"ContainerDied","Data":"6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb"}
Sep 15 09:46:33.459615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.459310    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f4884bd-pqwng" event={"ID":"1db095bd-035a-4d2b-b2e5-1d71ce848b97","Type":"ContainerDied","Data":"db8cc9952c596e93a3657b78f6cbf1b237f6cd962bcf9f695189e14156f98ca0"}
Sep 15 09:46:33.459615 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.459334    2571 scope.go:117] "RemoveContainer" containerID="6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb"
Sep 15 09:46:33.469256 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.469236    2571 scope.go:117] "RemoveContainer" containerID="6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb"
Sep 15 09:46:33.469548 ip-10-0-134-28 kubenswrapper[2571]: E0915 09:46:33.469527    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb\": container with ID starting with 6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb not found: ID does not exist" containerID="6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb"
Sep 15 09:46:33.469625 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.469561    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb"} err="failed to get container status \"6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb\": rpc error: code = NotFound desc = could not find container \"6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb\": container with ID starting with 6864a77fe83db74587d0e5484e202bffba0f93a66d4dc0aa902a36b3165f62eb not found: ID does not exist"
Sep 15 09:46:33.502903 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.502673    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f4884bd-pqwng"]
Sep 15 09:46:33.507491 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:33.507467    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f4884bd-pqwng"]
Sep 15 09:46:34.058599 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:34.058550    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db095bd-035a-4d2b-b2e5-1d71ce848b97" path="/var/lib/kubelet/pods/1db095bd-035a-4d2b-b2e5-1d71ce848b97/volumes"
Sep 15 09:46:39.510756 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:39.510715    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:45.409563 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:45.409507    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:48.178957 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:48.178909    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:46:48.186375 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:48.186351    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:46:52.604473 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:52.604437    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fhrnp"]
Sep 15 09:46:54.886951 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:46:54.886919    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748b84f758-jw6tb_cc6c6ab9-2d03-45e1-83e2-29f9e42374e2/authorino/0.log"
Sep 15 09:47:00.637482 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:00.637454    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dcfd7f98-6jqhk_e3b9355e-a720-4212-881d-58e0b6de970c/copy-manifests/0.log"
Sep 15 09:47:00.644039 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:00.644012    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dcfd7f98-6jqhk_e3b9355e-a720-4212-881d-58e0b6de970c/manager/0.log"
Sep 15 09:47:01.219579 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:01.219546    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-94fbc46d8-4nlg5_ffc3e241-2107-465a-a548-ed0b6ad5ece4/maas-api/0.log"
Sep 15 09:47:02.397008 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:02.396974    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748b84f758-jw6tb_cc6c6ab9-2d03-45e1-83e2-29f9e42374e2/authorino/0.log"
Sep 15 09:47:02.965277 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:02.965238    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fhrnp_fd294ba5-6ad4-49a3-8ed2-e08eea8689f3/limitador/0.log"
Sep 15 09:47:03.687773 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:03.687733    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b88c986-7p5bl_68dc7d0e-069a-401c-91f1-d0fc776c4569/kube-auth-proxy/0.log"
Sep 15 09:47:03.913862 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:03.913818    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/1.log"
Sep 15 09:47:04.262938 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:04.262909    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c4fbf958-wdqdc_f996fced-148b-4c99-9ad7-b885c09f618c/router/0.log"
Sep 15 09:47:05.184299 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:05.184268    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jdl7t_0b0f4329-cd5b-453f-83b1-89b180df9c0d/main/0.log"
Sep 15 09:47:05.298085 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:05.298054    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-q25hl_7f400c0e-b16e-4698-bb51-e0e08c821ebe/main/0.log"
Sep 15 09:47:11.122895 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:11.122791    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:47:11.188424 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:11.188391    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log" limit=1024
Sep 15 09:47:14.115588 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:14.115547    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dqkpv_22a36c08-dd46-4c1c-85a6-d53a59b56d41/global-pull-secret-syncer/0.log"
Sep 15 09:47:14.245789 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:14.245748    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-748rj_931e8042-1c1c-4a89-9009-81d67154d70b/konnectivity-agent/0.log"
Sep 15 09:47:14.339159 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:14.339124    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-28.ec2.internal_ff78554c81b115a40bbe63702b1bfba3/haproxy/0.log"
Sep 15 09:47:19.035286 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:19.035246    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-748b84f758-jw6tb_cc6c6ab9-2d03-45e1-83e2-29f9e42374e2/authorino/0.log"
Sep 15 09:47:19.217539 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:19.217501    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fhrnp_fd294ba5-6ad4-49a3-8ed2-e08eea8689f3/limitador/0.log"
Sep 15 09:47:21.099743 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.099713    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fn6dl_f00af179-f92d-4591-a7fe-41397aa3c5b1/kube-state-metrics/0.log"
Sep 15 09:47:21.122817 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.122783    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fn6dl_f00af179-f92d-4591-a7fe-41397aa3c5b1/kube-rbac-proxy-main/0.log"
Sep 15 09:47:21.147557 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.147530    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fn6dl_f00af179-f92d-4591-a7fe-41397aa3c5b1/kube-rbac-proxy-self/0.log"
Sep 15 09:47:21.193974 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.193941    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57f6c54bd6-fhnvc_9a66d263-45db-41fb-8896-fc3dc7141efa/metrics-server/0.log"
Sep 15 09:47:21.280793 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.280711    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cw7w_33113bd9-0d82-42ea-925a-1150385869eb/node-exporter/0.log"
Sep 15 09:47:21.308493 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.308461    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cw7w_33113bd9-0d82-42ea-925a-1150385869eb/kube-rbac-proxy/0.log"
Sep 15 09:47:21.368567 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.368532    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2cw7w_33113bd9-0d82-42ea-925a-1150385869eb/init-textfile/0.log"
Sep 15 09:47:21.716490 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.716458    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/prometheus/0.log"
Sep 15 09:47:21.742645 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.742614    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/config-reloader/0.log"
Sep 15 09:47:21.769438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.769408    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/thanos-sidecar/0.log"
Sep 15 09:47:21.792820 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.792796    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/kube-rbac-proxy-web/0.log"
Sep 15 09:47:21.819859 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.819770    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/kube-rbac-proxy/0.log"
Sep 15 09:47:21.844434 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.844401    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/kube-rbac-proxy-thanos/0.log"
Sep 15 09:47:21.867157 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:21.867127    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3842aabf-ddf6-4aba-ac93-6c3623eb6544/init-config-reloader/0.log"
Sep 15 09:47:22.097221 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:22.097194    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d8ddb5c99-wg29g_ff455cd0-448b-435d-b3f8-cfe300a9d9ab/thanos-query/0.log"
Sep 15 09:47:22.129920 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:22.129887    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d8ddb5c99-wg29g_ff455cd0-448b-435d-b3f8-cfe300a9d9ab/kube-rbac-proxy-web/0.log"
Sep 15 09:47:22.154460 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:22.154426    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d8ddb5c99-wg29g_ff455cd0-448b-435d-b3f8-cfe300a9d9ab/kube-rbac-proxy/0.log"
Sep 15 09:47:22.179925 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:22.179898    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d8ddb5c99-wg29g_ff455cd0-448b-435d-b3f8-cfe300a9d9ab/prom-label-proxy/0.log"
Sep 15 09:47:22.203616 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:22.203587    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d8ddb5c99-wg29g_ff455cd0-448b-435d-b3f8-cfe300a9d9ab/kube-rbac-proxy-rules/0.log"
Sep 15 09:47:22.229402 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:22.229372    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d8ddb5c99-wg29g_ff455cd0-448b-435d-b3f8-cfe300a9d9ab/kube-rbac-proxy-metrics/0.log"
Sep 15 09:47:23.637797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:23.637763    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-fw5zp_caa97be3-bc67-45b4-af76-1eab51a65a35/networking-console-plugin/0.log"
Sep 15 09:47:24.270915 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.270877    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/2.log"
Sep 15 09:47:24.274913 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.274886    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-87nft_e1d4492b-ba6e-4e83-9d63-6b89747de67e/console-operator/3.log"
Sep 15 09:47:24.836873 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.836822    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"]
Sep 15 09:47:24.837259 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.837201    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db095bd-035a-4d2b-b2e5-1d71ce848b97" containerName="maas-api"
Sep 15 09:47:24.837259 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.837212    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db095bd-035a-4d2b-b2e5-1d71ce848b97" containerName="maas-api"
Sep 15 09:47:24.837331 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.837276    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db095bd-035a-4d2b-b2e5-1d71ce848b97" containerName="maas-api"
Sep 15 09:47:24.840278 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.840254    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:24.843852 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.843812    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mwkv\"/\"openshift-service-ca.crt\""
Sep 15 09:47:24.843988 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.843835    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mwkv\"/\"default-dockercfg-r2jw9\""
Sep 15 09:47:24.844960 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.844944    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mwkv\"/\"kube-root-ca.crt\""
Sep 15 09:47:24.852194 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.852162    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"]
Sep 15 09:47:24.862033 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.862005    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-jnbzd_d3226dfa-ddd2-45f6-96b9-b03a879ba935/download-server/0.log"
Sep 15 09:47:24.958918 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.958865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-sys\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:24.959127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.958954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-lib-modules\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:24.959127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.958973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4mm8\" (UniqueName: \"kubernetes.io/projected/3bfd9fee-2442-45c3-a81b-41133825a2ca-kube-api-access-p4mm8\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:24.959127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.959001    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-proc\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:24.959127 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:24.959016    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-podres\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060286 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-proc\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060286 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060288    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-podres\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-sys\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060383    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-proc\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-lib-modules\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4mm8\" (UniqueName: \"kubernetes.io/projected/3bfd9fee-2442-45c3-a81b-41133825a2ca-kube-api-access-p4mm8\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060492 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060440    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-sys\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060509    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-podres\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.060670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.060567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bfd9fee-2442-45c3-a81b-41133825a2ca-lib-modules\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.068670 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.068642    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4mm8\" (UniqueName: \"kubernetes.io/projected/3bfd9fee-2442-45c3-a81b-41133825a2ca-kube-api-access-p4mm8\") pod \"perf-node-gather-daemonset-n7frl\" (UID: \"3bfd9fee-2442-45c3-a81b-41133825a2ca\") " pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.150874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.150745    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.443445 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.443420    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"]
Sep 15 09:47:25.445472 ip-10-0-134-28 kubenswrapper[2571]: W0915 09:47:25.445440    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3bfd9fee_2442_45c3_a81b_41133825a2ca.slice/crio-315346e9e9958fe7bf159487e3fcf9914e8f4e000806521bbc80a63cff49fd66 WatchSource:0}: Error finding container 315346e9e9958fe7bf159487e3fcf9914e8f4e000806521bbc80a63cff49fd66: Status 404 returned error can't find the container with id 315346e9e9958fe7bf159487e3fcf9914e8f4e000806521bbc80a63cff49fd66
Sep 15 09:47:25.709750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.709655    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl" event={"ID":"3bfd9fee-2442-45c3-a81b-41133825a2ca","Type":"ContainerStarted","Data":"016121e0dd7cee95c3114139dcd4a79eb697409aee4d1222b3c9d2ec27737d1e"}
Sep 15 09:47:25.709750 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.709707    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl" event={"ID":"3bfd9fee-2442-45c3-a81b-41133825a2ca","Type":"ContainerStarted","Data":"315346e9e9958fe7bf159487e3fcf9914e8f4e000806521bbc80a63cff49fd66"}
Sep 15 09:47:25.709977 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.709852    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:25.729908 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:25.729852    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl" podStartSLOduration=1.729814532 podStartE2EDuration="1.729814532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 09:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 09:47:25.728390064 +0000 UTC m=+2438.187679607" watchObservedRunningTime="2026-09-15 09:47:25.729814532 +0000 UTC m=+2438.189104073"
Sep 15 09:47:26.954174 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:26.954143    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vc4p7_795e1c32-4def-4adb-abce-c118db86156f/dns/0.log"
Sep 15 09:47:26.976438 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:26.976407    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vc4p7_795e1c32-4def-4adb-abce-c118db86156f/kube-rbac-proxy/0.log"
Sep 15 09:47:27.005212 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:27.005188    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxw6x_ef628e49-f57f-43ab-a6df-2eed9e923881/dns-node-resolver/0.log"
Sep 15 09:47:27.652752 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:27.652718    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2494n_10e5b976-0d90-4ed2-a797-274776807382/node-ca/0.log"
Sep 15 09:47:29.602970 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:29.602945    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b88c986-7p5bl_68dc7d0e-069a-401c-91f1-d0fc776c4569/kube-auth-proxy/0.log"
Sep 15 09:47:29.659767 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:29.659740    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/0.log"
Sep 15 09:47:29.662797 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:29.662775    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/1.log"
Sep 15 09:47:29.776423 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:29.776378    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c4fbf958-wdqdc_f996fced-148b-4c99-9ad7-b885c09f618c/router/0.log"
Sep 15 09:47:30.652583 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:30.652544    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qwp25_fdc7ebe9-46d4-4905-af95-2ae799a30157/serve-healthcheck-canary/0.log"
Sep 15 09:47:31.345030 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:31.345001    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wfwmf_0a59c6c9-0815-4d7e-9e51-32da9d62bbd2/kube-rbac-proxy/0.log"
Sep 15 09:47:31.366509 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:31.366482    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wfwmf_0a59c6c9-0815-4d7e-9e51-32da9d62bbd2/exporter/0.log"
Sep 15 09:47:31.388865 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:31.388839    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wfwmf_0a59c6c9-0815-4d7e-9e51-32da9d62bbd2/extractor/0.log"
Sep 15 09:47:31.725386 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:31.725316    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mwkv/perf-node-gather-daemonset-n7frl"
Sep 15 09:47:34.370301 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:34.370271    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dcfd7f98-6jqhk_e3b9355e-a720-4212-881d-58e0b6de970c/manager/0.log"
Sep 15 09:47:34.392867 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:34.392821    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77dcfd7f98-6jqhk_e3b9355e-a720-4212-881d-58e0b6de970c/copy-manifests/0.log"
Sep 15 09:47:35.724730 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:35.724701    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b88c986-7p5bl_68dc7d0e-069a-401c-91f1-d0fc776c4569/kube-auth-proxy/0.log"
Sep 15 09:47:35.768874 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:35.768819    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/0.log"
Sep 15 09:47:35.771642 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:35.771608    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/1.log"
Sep 15 09:47:35.853310 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:35.853251    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c4fbf958-wdqdc_f996fced-148b-4c99-9ad7-b885c09f618c/router/0.log"
Sep 15 09:47:35.938281 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:35.938191    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f667c6f8-66z7p_505e8ef8-30b0-462f-ad41-f747542f37ac/manager/0.log"
Sep 15 09:47:38.271853 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:38.271806    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b88c986-7p5bl_68dc7d0e-069a-401c-91f1-d0fc776c4569/kube-auth-proxy/0.log"
Sep 15 09:47:38.306921 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:38.306887    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/0.log"
Sep 15 09:47:38.310057 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:38.310035    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cxzvg_f59fb6b2-5998-411a-9fc6-cd26950366a0/istio-proxy/1.log"
Sep 15 09:47:38.393306 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:38.393275    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c4fbf958-wdqdc_f996fced-148b-4c99-9ad7-b885c09f618c/router/0.log"
Sep 15 09:47:40.982166 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:40.982132    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-dwb4h_51c0546e-33df-4ae4-be9c-04e9a4dca2fc/migrator/0.log"
Sep 15 09:47:41.006251 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:41.006223    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-dwb4h_51c0546e-33df-4ae4-be9c-04e9a4dca2fc/graceful-termination/0.log"
Sep 15 09:47:41.369297 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:41.369265    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-cj2td_31b4dd4b-c91c-4e10-87cf-7dd643e2edee/kube-storage-version-migrator-operator/1.log"
Sep 15 09:47:41.370182 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:41.370164    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-cj2td_31b4dd4b-c91c-4e10-87cf-7dd643e2edee/kube-storage-version-migrator-operator/0.log"
Sep 15 09:47:42.352869 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.352821    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/kube-multus-additional-cni-plugins/0.log"
Sep 15 09:47:42.376071 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.376041    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/egress-router-binary-copy/0.log"
Sep 15 09:47:42.396890 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.396860    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/cni-plugins/0.log"
Sep 15 09:47:42.419999 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.419968    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/bond-cni-plugin/0.log"
Sep 15 09:47:42.442346 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.442315    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/routeoverride-cni/0.log"
Sep 15 09:47:42.463714 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.463690    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/whereabouts-cni-bincopy/0.log"
Sep 15 09:47:42.484554 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.484515    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgrjz_32565e69-75c9-48aa-a026-ed63498f2fa8/whereabouts-cni/0.log"
Sep 15 09:47:42.975995 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:42.975957    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vh8mh_ad9d08e1-762d-46c7-bf7c-0723d12e33cb/kube-multus/0.log"
Sep 15 09:47:43.088087 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:43.088057    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d8h8m_84527138-dafc-4435-b121-9215368bcb16/network-metrics-daemon/0.log"
Sep 15 09:47:43.108145 ip-10-0-134-28 kubenswrapper[2571]: I0915 09:47:43.108119    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d8h8m_84527138-dafc-4435-b121-9215368bcb16/kube-rbac-proxy/0.log"