Sep 15 18:48:52.105960 ip-10-0-143-252 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 18:48:52.105969 ip-10-0-143-252 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 18:48:52.105976 ip-10-0-143-252 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 18:48:52.106272 ip-10-0-143-252 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 18:49:03.267055 ip-10-0-143-252 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 18:49:03.267071 ip-10-0-143-252 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 66a88e7829cf4728bc31583055f59025 --
Sep 15 18:50:47.740921 ip-10-0-143-252 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 18:50:48.084355 ip-10-0-143-252 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:48.084355 ip-10-0-143-252 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 18:50:48.084355 ip-10-0-143-252 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:48.084355 ip-10-0-143-252 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 18:50:48.084355 ip-10-0-143-252 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 18:50:48.085974 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.085890    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 18:50:48.091274 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091257    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:48.091274 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091274    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091277    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091280    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091284    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091287    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091290    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091292    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091295    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091298    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091300    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091303    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091307    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091311    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091315    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091318    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091321    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091324    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091327    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091329    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:48.091341 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091333    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091336    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091339    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091341    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091344    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091347    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091350    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091354    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091356    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091359    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091362    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091364    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091367    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091370    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091373    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091375    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091378    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091381    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091383    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091386    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:48.091832 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091388    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091390    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091393    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091395    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091399    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091404    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091408    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091410    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091413    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091416    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091418    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091421    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091424    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091426    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091430    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091432    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091435    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091437    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091440    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091442    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:48.092342 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091445    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091447    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091450    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091453    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091455    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091458    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091460    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091463    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091466    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091468    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091471    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091474    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091476    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091479    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091481    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091484    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091487    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091490    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091494    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091497    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:48.092891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091499    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091502    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091505    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091507    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091510    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091512    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091518    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091521    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091524    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091527    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091530    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091533    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091536    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091538    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091541    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091544    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091547    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091550    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.091552    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:48.093413 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092339    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092346    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092350    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092353    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092356    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092358    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092361    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092364    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092367    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092369    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092373    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092378    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092393    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092397    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092400    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092403    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092406    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092409    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092411    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:48.093891 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092414    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092417    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092420    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092422    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092425    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092428    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092430    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092434    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092437    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092439    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092442    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092445    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092447    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092451    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092454    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092457    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092461    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092463    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092466    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092468    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:48.094372 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092470    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092473    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092475    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092479    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092482    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092485    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092493    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092495    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092498    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092502    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092505    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092507    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092510    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092513    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092515    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092518    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092521    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092523    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092526    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092528    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:48.094879 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092531    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092533    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092535    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092538    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092540    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092543    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092545    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092548    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092550    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092553    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092555    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092558    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092560    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092563    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092565    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092568    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092571    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092573    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092576    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092583    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:48.095371 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092586    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092589    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092591    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092594    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092596    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092599    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092602    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092604    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092606    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092624    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092626    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092629    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092631    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092634    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092637    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092639    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092643    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092646    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092649    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:48.095877 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.092651    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093042    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093047    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093050    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093052    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093056    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093058    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093061    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093064    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093066    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093069    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093071    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093074    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093077    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093079    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093081    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093085    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093088    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093092    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:48.096362 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093096    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093099    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093102    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093105    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093108    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093111    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093114    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093116    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093119    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093122    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093126    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093130    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093133    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093136    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093138    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093141    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093144    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093146    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093148    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093151    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:48.096846 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093154    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093157    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093160    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093162    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093165    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093167    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093170    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093172    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093175    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093178    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093180    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093183    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093185    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093188    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093190    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093193    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093196    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093198    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093201    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:48.097343 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093203    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093205    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093208    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093210    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093213    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093217    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093220    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093222    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093225    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093227    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093229    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093232    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093234    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093238    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093241    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093244    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093247    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093250    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093253    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093255    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:48.097888 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093259    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093264    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093267    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093269    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093272    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093274    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093277    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093280    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093282    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093284    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093287    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093290    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093292    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093295    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093297    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093300    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093302    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093306    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093309    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093311    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:48.098481 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093314    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.093316    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093391    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093422    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093429    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093434    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093439    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093442    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093446    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093450    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093454    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093457    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093460    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093465    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093468    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093471    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093474    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093477    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093480    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093483    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093487    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093490    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093493    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093496    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 18:50:48.098997 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093499    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093503    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093506    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093509    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093512    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093516    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093519    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093522    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093525    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093528    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093532    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093535    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093538    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093541    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093544    2584 flags.go:64] FLAG: --enable-server="true"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093547    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093551    2584 flags.go:64] FLAG: --event-burst="100"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093554    2584 flags.go:64] FLAG: --event-qps="50"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093557    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093560    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093563    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093567    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093571    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093574    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093577    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 15 18:50:48.099578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093580    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093583    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093586    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093589    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093592    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093595    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093598    2584 flags.go:64] FLAG: --feature-gates=""
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093602    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093605    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093621    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093625    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093628    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093632    2584 flags.go:64] FLAG: --help="false"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093635    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093638    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093641    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093644    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093648    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093651    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093654    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093657    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093660    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093663    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 18:50:48.100415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093666    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093669    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093672    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093674    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093677    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093681    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093684    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093688    2584 flags.go:64] FLAG: --lock-file=""
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093691    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093694    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093697    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093702    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093705    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093708    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093711    2584 flags.go:64] FLAG: --logging-format="text"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093714    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093718    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093721    2584 flags.go:64] FLAG: --manifest-url=""
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093724    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093728    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093731    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093736    2584 flags.go:64] FLAG: --max-pods="110"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093739    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093742    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093745    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 18:50:48.101228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093749    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093752    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093755    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093757    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093764    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093767    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093770    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093773    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093776    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093781    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093784    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093791    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093794    2584 flags.go:64] FLAG: --port="10250"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093797    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093800    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0452e5a8f4b5ce093"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093804    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093807    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093810    2584 flags.go:64] FLAG: --register-node="true"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093813    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093816    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093819    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093822    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093825    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093828    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093831    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 18:50:48.101938 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093834    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093837    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093840    2584 flags.go:64] FLAG: --runonce="false"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093843    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093846    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093849    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093853    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093856    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093859    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093862    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093865    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093868    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093871    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093873    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093876    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093880    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093882    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093885    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093890    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093893    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093897    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093901    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093904    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093907    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093910    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 18:50:48.102553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093913    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093916    2584 flags.go:64] FLAG: --v="2"
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093920    2584 flags.go:64] FLAG: --version="false"
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093924    2584 flags.go:64] FLAG: --vmodule=""
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093933    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.093937    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094055    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094059    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094061    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094064    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094068    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094070    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094073    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094075    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094078    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094081    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094083    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094086    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094088    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094091    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094094    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:48.103183 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094096    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094099    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094101    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094104    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094106    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094109    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094113    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094116    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094119    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094122    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094125    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094128    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094130    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094133    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094135    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094138    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094140    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094143    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094145    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:48.103724 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094148    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094151    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094154    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094156    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094159    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094161    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094165    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094169    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094172    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094175    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094178    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094181    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094184    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094187    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094190    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094193    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094195    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094198    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094200    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:48.104207 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094204    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094207    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094209    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094213    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094217    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094221    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094223    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094226    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094228    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094231    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094233    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094238    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094241    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094243    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094246    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094249    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094252    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094255    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094258    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094261    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:48.104706 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094264    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094266    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094269    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094272    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094275    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094278    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094281    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094283    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094286    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094288    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094290    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094293    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094297    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094299    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094302    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094304    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094307    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094310    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094313    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094315    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:48.105199 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094318    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094321    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094323    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094326    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094329    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.094331    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.094339    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.102935    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.102951    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.103641    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.103652    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103711    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103717    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103722    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:48.105757 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103725    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103728    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103733    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103736    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103740    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103743    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103746    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103750    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103752    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103755    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103758    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103760    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103763    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103766    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103768    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103771    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103773    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103781    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103784    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:48.106111 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103787    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103790    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103792    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103795    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103798    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103800    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103803    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103807    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103810    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103813    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103815    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103818    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103821    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103823    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103826    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103828    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103831    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103834    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103836    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:48.106584 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103839    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103842    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103844    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103847    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103850    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103853    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103856    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103858    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103861    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103864    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103867    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103869    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103872    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103876    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103878    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103881    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103883    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103886    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103889    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103891    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:48.107086 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103894    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103897    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103900    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103903    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103907    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103911    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103914    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103917    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103920    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103923    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103926    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103928    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103931    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103933    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103936    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103939    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103941    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103944    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103947    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103950    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:48.107629 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103953    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103955    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103958    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103961    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103965    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103967    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103970    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103973    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103976    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103978    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103980    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103983    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103986    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103988    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103991    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103993    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103996    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:48.108138 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.103998    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.104003    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104127    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104133    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104136    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104139    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104142    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104145    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104147    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104151    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104155    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104158    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104161    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104163    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104166    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 18:50:48.108637 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104169    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104172    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104175    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104178    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104181    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104183    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104186    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104189    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104192    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104196    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104199    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104202    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104205    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104208    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104211    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104213    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104217    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104220    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104223    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104225    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 18:50:48.109265 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104228    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104230    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104233    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104235    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104238    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104240    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104243    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104247    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104249    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104252    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104254    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104257    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104260    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104262    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104265    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104268    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104271    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104273    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104276    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104279    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 18:50:48.109974 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104281    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104284    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104286    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104289    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104291    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104294    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104296    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104299    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104301    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104304    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104306    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104308    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104311    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104314    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104317    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104319    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104322    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104324    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104326    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104329    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 18:50:48.110694 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104332    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104334    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104336    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104339    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104342    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104345    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104347    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104350    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104353    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104355    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104358    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104362    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104365    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104368    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104370    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104373    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104375    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104378    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104380    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 18:50:48.111256 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104383    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104385    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104388    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104390    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104393    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104395    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:48.104398    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.104403    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.104514    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.107332    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.108391    2584 server.go:1019] "Starting client certificate rotation"
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.108509    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.108553    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 18:50:48.111753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.110677    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:48.113403 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.113386    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 18:50:48.121218 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.121201    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 15 18:50:48.124002 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.123985    2584 log.go:25] "Validated CRI v1 image API"
Sep 15 18:50:48.124819 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.124804    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 18:50:48.128565 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.128544    2584 fs.go:135] Filesystem UUIDs: map[1361fee8-7fd4-4f14-991c-0e34d915d015:/dev/nvme0n1p3 175a3904-b1ee-4875-9d52-b8754eab889b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 15 18:50:48.128658 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.128563    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 18:50:48.135450 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.135337    2584 manager.go:216] Machine: {Timestamp:2026-09-15 18:50:48.134261004 +0000 UTC m=+0.288629463 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099514 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21a409269e79f10d4b73a70c8fb748 SystemUUID:ec21a409-269e-79f1-0d4b-73a70c8fb748 BootID:66a88e78-29cf-4728-bc31-583055f59025 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:65:ff:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:65:ff:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:b4:40:ed:28:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 18:50:48.135450 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.135438    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 18:50:48.135605 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.135525    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 18:50:48.138279 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.138256    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 18:50:48.138408 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.138281    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 18:50:48.139010 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.138997    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 18:50:48.139010 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.139009    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 18:50:48.139110 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.139017    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:48.139596 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.139587    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 18:50:48.140700 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.140680    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:48.140837 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.140824    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 18:50:48.144791 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144779    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:48.144910 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144900    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 18:50:48.144951 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144930    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 18:50:48.144951 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144943    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 18:50:48.145006 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144968    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 18:50:48.145006 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144981    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 15 18:50:48.145006 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.144996    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 18:50:48.145334 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.145324    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:48.145384 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.145342    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 18:50:48.147142 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.147127    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 18:50:48.149545 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.149531    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 18:50:48.149593 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.149562    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 18:50:48.151371 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151358    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 18:50:48.151406 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151380    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 18:50:48.151406 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151389    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 18:50:48.151406 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151396    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151409    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151417    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151439    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151447    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151457    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151465    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151478    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 18:50:48.151506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.151486    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 18:50:48.152199 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.152188    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 18:50:48.152231 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.152201    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 18:50:48.155561 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.155540    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 18:50:48.155889 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.155876    2584 server.go:1268] "Started kubelet"
Sep 15 18:50:48.155993 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.155950    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 18:50:48.156073 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.156037    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 18:50:48.156117 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.156087    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 18:50:48.156361 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.156325    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 18:50:48.156420 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.156327    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 18:50:48.156836 ip-10-0-143-252 systemd[1]: Started Kubernetes Kubelet.
Sep 15 18:50:48.157259 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.157234    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 18:50:48.160367 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.160327    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gklrh"
Sep 15 18:50:48.160940 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.160016    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-252.ec2.internal.18d5932cef8359b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-252.ec2.internal,UID:ip-10-0-143-252.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-252.ec2.internal,},FirstTimestamp:2026-09-15 18:50:48.155847089 +0000 UTC m=+0.310215548,LastTimestamp:2026-09-15 18:50:48.155847089 +0000 UTC m=+0.310215548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-252.ec2.internal,}"
Sep 15 18:50:48.161029 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.160947    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 18:50:48.161224 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.161211    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 18:50:48.163245 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.163228    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:48.163718 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.163703    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 18:50:48.163866 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.163828    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.164344 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.164325    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 18:50:48.164344 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.164331    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 18:50:48.164451 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.164352    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 18:50:48.164491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.164461    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 18:50:48.164491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.164468    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 18:50:48.165130 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165115    2584 factory.go:55] Registering systemd factory
Sep 15 18:50:48.165207 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165134    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 15 18:50:48.165338 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165327    2584 factory.go:153] Registering CRI-O factory
Sep 15 18:50:48.165380 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165339    2584 factory.go:223] Registration of the crio container factory successfully
Sep 15 18:50:48.165412 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165400    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 18:50:48.165445 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165414    2584 factory.go:103] Registering Raw factory
Sep 15 18:50:48.165445 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.165425    2584 manager.go:1196] Started watching for new ooms in manager
Sep 15 18:50:48.165945 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.165915    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 18:50:48.166287 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.166260    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 18:50:48.166576 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.166562    2584 manager.go:349] Starting recovery of all containers
Sep 15 18:50:48.168456 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.168431    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gklrh"
Sep 15 18:50:48.178360 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.178245    2584 manager.go:354] Recovery completed
Sep 15 18:50:48.184287 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.184267    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.188119 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188094    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.188201 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188134    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.188201 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188155    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.188765 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188751    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 18:50:48.188765 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188758    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.188903 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188761    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 18:50:48.188903 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.188858    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 18:50:48.189482 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.189466    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.189552 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.189494    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.189552 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.189509    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.190971 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.190958    2584 policy_none.go:49] "None policy: Start"
Sep 15 18:50:48.191049 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.190976    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 18:50:48.191049 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.190989    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 18:50:48.192955 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.192942    2584 policy_none.go:47] "Start"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.225346    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226430    2584 manager.go:337] "Starting Device Plugin manager"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.226470    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226482    2584 server.go:87] "Starting device plugin registration server"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226552    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226643    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226677    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.226742    2584 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226911    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.226924    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.227061    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.227124    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.227131    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.228101    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.228150    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.240226 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.229087    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:48.327307 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.327272    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.327483 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.327314    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"]
Sep 15 18:50:48.327483 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.327392    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.328242 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328222    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.328350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328250    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.328350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328260    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.328350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328264    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.328350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328281    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.328350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328284    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.328350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.328290    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.330553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.330541    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.330671 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.330656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.330722 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.330684    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.331195 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.331177    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.331304 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.331205    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.331304 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.331178    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.331304 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.331218    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.331304 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.331232    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.331304 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.331245    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.333318 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.333302    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.333407 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.333335    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 18:50:48.333993 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.333979    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 18:50:48.334079 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.334005    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 18:50:48.334079 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.334020    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeHasSufficientPID"
Sep 15 18:50:48.336013 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.335937    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.336013 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.335962    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-252.ec2.internal\": node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.352344 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.352327    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.357130 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.357115    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-252.ec2.internal\" not found" node="ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.361550 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.361536    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-252.ec2.internal\" not found" node="ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.365168 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.365151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3003a6cfa64b8558b5eab03a3328f05d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"3003a6cfa64b8558b5eab03a3328f05d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.365237 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.365174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e01db16d59f3dba97f5f5f5aaef31d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-252.ec2.internal\" (UID: \"b7e01db16d59f3dba97f5f5f5aaef31d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.365237 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.365188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3003a6cfa64b8558b5eab03a3328f05d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"3003a6cfa64b8558b5eab03a3328f05d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.453114 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.453081    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.465757 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.465732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3003a6cfa64b8558b5eab03a3328f05d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"3003a6cfa64b8558b5eab03a3328f05d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.465814 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.465762    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e01db16d59f3dba97f5f5f5aaef31d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-252.ec2.internal\" (UID: \"b7e01db16d59f3dba97f5f5f5aaef31d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.465814 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.465779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3003a6cfa64b8558b5eab03a3328f05d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"3003a6cfa64b8558b5eab03a3328f05d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.465883 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.465847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3003a6cfa64b8558b5eab03a3328f05d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"3003a6cfa64b8558b5eab03a3328f05d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.465883 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.465856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3003a6cfa64b8558b5eab03a3328f05d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal\" (UID: \"3003a6cfa64b8558b5eab03a3328f05d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.465949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.465847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b7e01db16d59f3dba97f5f5f5aaef31d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-252.ec2.internal\" (UID: \"b7e01db16d59f3dba97f5f5f5aaef31d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.553594 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.553557    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.654368 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.654302    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.660495 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.660473    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.665593 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:48.665573    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:48.754438 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.754404    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.854968 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.854927    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:48.955565 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:48.955472    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.012379 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.012348    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:49.056245 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:49.056215    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.108694 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.108664    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 18:50:49.109142 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.108837    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:49.109142 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.108836    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 18:50:49.156797 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:49.156769    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.174206 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.174168    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 18:45:48 +0000 UTC" deadline="2028-04-05 12:47:20.628265432 +0000 UTC"
Sep 15 18:50:49.174206 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.174198    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13625h56m31.454072029s"
Sep 15 18:50:49.257040 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:49.257007    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.358092 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:49.358069    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.369396 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:49.369359    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3003a6cfa64b8558b5eab03a3328f05d.slice/crio-ca00e6ee6d5102cab5ba87d6e69db0a33a08d044cafab462a0502b1b9c510be8 WatchSource:0}: Error finding container ca00e6ee6d5102cab5ba87d6e69db0a33a08d044cafab462a0502b1b9c510be8: Status 404 returned error can't find the container with id ca00e6ee6d5102cab5ba87d6e69db0a33a08d044cafab462a0502b1b9c510be8
Sep 15 18:50:49.369849 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:49.369822    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e01db16d59f3dba97f5f5f5aaef31d.slice/crio-514794c645c36350fdf4cf75eda5626352b084233b66b2646cd4241f93b5e15a WatchSource:0}: Error finding container 514794c645c36350fdf4cf75eda5626352b084233b66b2646cd4241f93b5e15a: Status 404 returned error can't find the container with id 514794c645c36350fdf4cf75eda5626352b084233b66b2646cd4241f93b5e15a
Sep 15 18:50:49.374981 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.374964    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:50:49.462940 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:49.462890    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.535249 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.535167    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:49.563685 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:49.563652    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-252.ec2.internal\" not found"
Sep 15 18:50:49.660350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.660321    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:49.664491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.664471    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:49.672451 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.672428    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:49.673604 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.673590    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"
Sep 15 18:50:49.685047 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:49.685027    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 18:50:50.146105 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.146069    2584 apiserver.go:52] "Watching apiserver"
Sep 15 18:50:50.153534 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.153501    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 18:50:50.154069 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.154037    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ql8n6","openshift-multus/multus-zjntw","openshift-ovn-kubernetes/ovnkube-node-9qvxx","kube-system/konnectivity-agent-7zbgr","kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2","openshift-cluster-node-tuning-operator/tuned-hglmw","openshift-image-registry/node-ca-khm77","openshift-multus/network-metrics-daemon-x77j7","openshift-network-diagnostics/network-check-target-nz4mf","openshift-network-operator/iptables-alerter-f5fws","kube-system/global-pull-secret-syncer-jqw9v","openshift-dns/node-resolver-px5bd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal"]
Sep 15 18:50:50.156910 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.156761    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.159371 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.159343    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.159516 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.159370    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.160103 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.159662    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5rn97\""
Sep 15 18:50:50.160103 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.159991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.163339 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.163295    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.163588 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.163569    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 18:50:50.163918 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.163890    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8jbfc\""
Sep 15 18:50:50.164365 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.164345    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.164713 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.164539    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 18:50:50.165451 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.164943    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.165738 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.165703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.167341 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.167320    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 18:50:50.167440 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.167323    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.170277 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.170252    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.170917    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.171300    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.171309    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.171662    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-btcwk\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.172035    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2m2lq\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.172077    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.172315    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.172879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.172515    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 18:50:50.175284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.173926    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 18:50:50.175284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.174505    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.176207 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-modprobe-d\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176312 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-host\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176312 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176240    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.176312 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysconfig\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176312 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-var-lib-kubelet\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-tuned\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-tmp\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq75g\" (UniqueName: \"kubernetes.io/projected/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-kube-api-access-gq75g\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-var-lib-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176452    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-cni-bin\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176473    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.176537 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmk5\" (UniqueName: \"kubernetes.io/projected/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-kube-api-access-pfmk5\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.176983 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-kubelet\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.176983 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176705    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8tmqk\""
Sep 15 18:50:50.176983 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176729    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.177139 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177078    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.177139 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.177246 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.177169    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:50:50.177326 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177307    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qv9r\""
Sep 15 18:50:50.177424 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.176621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.177491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177444    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovnkube-script-lib\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.177491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/364d3196-120d-4ad1-b2b5-7c5d71b0351f-konnectivity-ca\") pod \"konnectivity-agent-7zbgr\" (UID: \"364d3196-120d-4ad1-b2b5-7c5d71b0351f\") " pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.177491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-kubernetes\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.177644 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-run\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.177644 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-lib-modules\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.177644 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-ovn\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.177821 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/364d3196-120d-4ad1-b2b5-7c5d71b0351f-agent-certs\") pod \"konnectivity-agent-7zbgr\" (UID: \"364d3196-120d-4ad1-b2b5-7c5d71b0351f\") " pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.177821 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c798e072-df04-4f79-a74b-ba6102067c23-iptables-alerter-script\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.177821 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177752    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-slash\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.177821 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177804    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-node-log\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177832    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-env-overrides\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysctl-conf\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-systemd-units\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177912    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-etc-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177937    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-log-socket\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.177984    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysctl-d\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.178023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178008    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-systemd\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-cni-netd\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovn-node-metrics-cert\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-systemd\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178155    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-sys\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178169    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c798e072-df04-4f79-a74b-ba6102067c23-host-slash\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncqr\" (UniqueName: \"kubernetes.io/projected/c798e072-df04-4f79-a74b-ba6102067c23-kube-api-access-nncqr\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178246    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-run-netns\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178414 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovnkube-config\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.178963 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.178462    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.179021 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.179005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.179469 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.179445    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 18:50:50.181143 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.181120    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.181605 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.181486    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlffw\""
Sep 15 18:50:50.181807 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.181767    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.182410 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.182389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.184080 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.184060    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:50.184180 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.184152    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:50:50.184773 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.184752    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rklgq\""
Sep 15 18:50:50.184978 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.184962    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 18:50:50.185268 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.185250    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 18:50:50.185484 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.185461    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 18:50:50.185682 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.185665    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 18:50:50.186852 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.186814    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.189148 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.189128    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t26b7\""
Sep 15 18:50:50.189252 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.189174    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 18:50:50.189569 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.189550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.189657 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.189622    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:50:50.189732 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.188976    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 18:50:50.199711 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.199664    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fznnb"
Sep 15 18:50:50.209790 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.209764    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fznnb"
Sep 15 18:50:50.215014 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.214990    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:50.234188 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.234130    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" event={"ID":"b7e01db16d59f3dba97f5f5f5aaef31d","Type":"ContainerStarted","Data":"514794c645c36350fdf4cf75eda5626352b084233b66b2646cd4241f93b5e15a"}
Sep 15 18:50:50.239160 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.239129    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" event={"ID":"3003a6cfa64b8558b5eab03a3328f05d","Type":"ContainerStarted","Data":"ca00e6ee6d5102cab5ba87d6e69db0a33a08d044cafab462a0502b1b9c510be8"}
Sep 15 18:50:50.265664 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.265631    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 18:50:50.278526 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:50.278692 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278545    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-var-lib-kubelet\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.278692 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-tmp\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.278692 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278602    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-var-lib-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.278692 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmk5\" (UniqueName: \"kubernetes.io/projected/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-kube-api-access-pfmk5\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.278692 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovnkube-script-lib\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278701    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9dz\" (UniqueName: \"kubernetes.io/projected/7aff6fb0-49d5-4729-b743-fdf79154f869-kube-api-access-ks9dz\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc2c1a6-b49e-406c-a839-ca526167e28a-cni-binary-copy\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-conf-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-kubernetes\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-lib-modules\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-ovn\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/364d3196-120d-4ad1-b2b5-7c5d71b0351f-agent-certs\") pod \"konnectivity-agent-7zbgr\" (UID: \"364d3196-120d-4ad1-b2b5-7c5d71b0351f\") " pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgr75\" (UniqueName: \"kubernetes.io/projected/067e10fb-5b75-446b-bc7c-7a7967b484ce-kube-api-access-wgr75\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278914    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-socket-dir-parent\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c798e072-df04-4f79-a74b-ba6102067c23-iptables-alerter-script\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.278979 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-slash\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.278994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-node-log\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279018    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-env-overrides\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjgm\" (UniqueName: \"kubernetes.io/projected/5fbf53cb-b32a-4e88-b5c1-2959df28a398-kube-api-access-8pjgm\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c258100-5235-4aee-b9a7-7e97267b0cb0-host\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279129    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c258100-5235-4aee-b9a7-7e97267b0cb0-serviceca\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-netns\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279159    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-var-lib-kubelet\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279180    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysctl-conf\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-systemd-units\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-log-socket\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279267    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5fbf53cb-b32a-4e88-b5c1-2959df28a398-hosts-file\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-cni-multus\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279319    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279339    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-hostroot\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.279645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysctl-d\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-systemd\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-cni-netd\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbf53cb-b32a-4e88-b5c1-2959df28a398-tmp-dir\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglml\" (UniqueName: \"kubernetes.io/projected/9c258100-5235-4aee-b9a7-7e97267b0cb0-kube-api-access-zglml\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279529    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-systemd\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-sys\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c798e072-df04-4f79-a74b-ba6102067c23-host-slash\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279646    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncqr\" (UniqueName: \"kubernetes.io/projected/c798e072-df04-4f79-a74b-ba6102067c23-kube-api-access-nncqr\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279720    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53c11c0f-bbb6-49d8-896e-d17c292a1644-dbus\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-cni-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279775    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nqs2\" (UniqueName: \"kubernetes.io/projected/cfc2c1a6-b49e-406c-a839-ca526167e28a-kube-api-access-8nqs2\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-modprobe-d\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.280544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279862    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279890    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-device-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-cnibin\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-multus-certs\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysconfig\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.279995    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-tuned\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq75g\" (UniqueName: \"kubernetes.io/projected/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-kube-api-access-gq75g\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-cni-bin\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/364d3196-120d-4ad1-b2b5-7c5d71b0351f-konnectivity-ca\") pod \"konnectivity-agent-7zbgr\" (UID: \"364d3196-120d-4ad1-b2b5-7c5d71b0351f\") " pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-system-cni-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-cni-bin\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-kubelet\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280190    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-etc-kubernetes\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-system-cni-dir\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cnibin\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cni-binary-copy\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.281383 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-run\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280358    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280386    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlsn\" (UniqueName: \"kubernetes.io/projected/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-kube-api-access-6qlsn\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-etc-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280466    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280534    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-kubelet\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-daemon-config\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovn-node-metrics-cert\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280642    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-k8s-cni-cncf-io\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-run-netns\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovnkube-config\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280743    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53c11c0f-bbb6-49d8-896e-d17c292a1644-kubelet-config\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.282267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-host\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280893    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.280959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysconfig\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281025    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-run\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281103    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-etc-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281254    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-node-log\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-slash\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c798e072-df04-4f79-a74b-ba6102067c23-iptables-alerter-script\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-env-overrides\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysctl-d\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-systemd\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281861    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-cni-netd\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-sysctl-conf\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-systemd-units\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.281961    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-log-socket\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282033    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-systemd\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-sys\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.283949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c798e072-df04-4f79-a74b-ba6102067c23-host-slash\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282142    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-modprobe-d\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282454    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-cni-bin\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-var-lib-openvswitch\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282909    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovnkube-config\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.282999    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-run-netns\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283058    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/364d3196-120d-4ad1-b2b5-7c5d71b0351f-konnectivity-ca\") pod \"konnectivity-agent-7zbgr\" (UID: \"364d3196-120d-4ad1-b2b5-7c5d71b0351f\") " pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-kubelet\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283214    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-host\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-kubernetes\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-run-ovn\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-lib-modules\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.285064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.283636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovnkube-script-lib\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.287310 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.287280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-tmp\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.287596 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.287547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/364d3196-120d-4ad1-b2b5-7c5d71b0351f-agent-certs\") pod \"konnectivity-agent-7zbgr\" (UID: \"364d3196-120d-4ad1-b2b5-7c5d71b0351f\") " pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.289423 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.289397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-ovn-node-metrics-cert\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.290878 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.290847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-etc-tuned\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.291417 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.291394    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncqr\" (UniqueName: \"kubernetes.io/projected/c798e072-df04-4f79-a74b-ba6102067c23-kube-api-access-nncqr\") pod \"iptables-alerter-f5fws\" (UID: \"c798e072-df04-4f79-a74b-ba6102067c23\") " pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.293059 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.293029    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq75g\" (UniqueName: \"kubernetes.io/projected/39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20-kube-api-access-gq75g\") pod \"tuned-hglmw\" (UID: \"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20\") " pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.293147 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.293095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmk5\" (UniqueName: \"kubernetes.io/projected/9da97d55-3907-44e9-bb0c-b5d1c044bfb1-kube-api-access-pfmk5\") pod \"ovnkube-node-9qvxx\" (UID: \"9da97d55-3907-44e9-bb0c-b5d1c044bfb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.381855 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-system-cni-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381864    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-cni-bin\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-etc-kubernetes\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-system-cni-dir\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381946    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cnibin\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-system-cni-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cni-binary-copy\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.381972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-cni-bin\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382027 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382032    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cnibin\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-system-cni-dir\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlsn\" (UniqueName: \"kubernetes.io/projected/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-kube-api-access-6qlsn\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-etc-kubernetes\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382146    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-kubelet\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-daemon-config\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382307    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-kubelet\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382309    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-k8s-cni-cncf-io\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382346    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-k8s-cni-cncf-io\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53c11c0f-bbb6-49d8-896e-d17c292a1644-kubelet-config\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.382438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382434    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9dz\" (UniqueName: \"kubernetes.io/projected/7aff6fb0-49d5-4729-b743-fdf79154f869-kube-api-access-ks9dz\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.382440    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc2c1a6-b49e-406c-a839-ca526167e28a-cni-binary-copy\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-conf-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.382532    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:50.88250925 +0000 UTC m=+3.036877731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382539    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-conf-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382555    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgr75\" (UniqueName: \"kubernetes.io/projected/067e10fb-5b75-446b-bc7c-7a7967b484ce-kube-api-access-wgr75\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-socket-dir-parent\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382614    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/53c11c0f-bbb6-49d8-896e-d17c292a1644-kubelet-config\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382625    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjgm\" (UniqueName: \"kubernetes.io/projected/5fbf53cb-b32a-4e88-b5c1-2959df28a398-kube-api-access-8pjgm\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c258100-5235-4aee-b9a7-7e97267b0cb0-host\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c258100-5235-4aee-b9a7-7e97267b0cb0-serviceca\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-netns\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382746    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cni-binary-copy\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5fbf53cb-b32a-4e88-b5c1-2959df28a398-hosts-file\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.383309 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-cni-multus\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-hostroot\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbf53cb-b32a-4e88-b5c1-2959df28a398-tmp-dir\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglml\" (UniqueName: \"kubernetes.io/projected/9c258100-5235-4aee-b9a7-7e97267b0cb0-kube-api-access-zglml\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382980    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-daemon-config\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53c11c0f-bbb6-49d8-896e-d17c292a1644-dbus\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383066    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-cni-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383104    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5fbf53cb-b32a-4e88-b5c1-2959df28a398-hosts-file\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nqs2\" (UniqueName: \"kubernetes.io/projected/cfc2c1a6-b49e-406c-a839-ca526167e28a-kube-api-access-8nqs2\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-var-lib-cni-multus\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383170    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-device-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383185    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-hostroot\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.384281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383195    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-cnibin\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-multus-certs\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.383294    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-multus-certs\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.383344    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:50.883328299 +0000 UTC m=+3.037696765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383416    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-socket-dir-parent\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383516    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/53c11c0f-bbb6-49d8-896e-d17c292a1644-dbus\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfc2c1a6-b49e-406c-a839-ca526167e28a-cni-binary-copy\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c258100-5235-4aee-b9a7-7e97267b0cb0-host\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-multus-cni-dir\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383625    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-host-run-netns\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383714    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfc2c1a6-b49e-406c-a839-ca526167e28a-cnibin\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383954    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-device-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.382856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/067e10fb-5b75-446b-bc7c-7a7967b484ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.383963    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbf53cb-b32a-4e88-b5c1-2959df28a398-tmp-dir\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.385109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.384036    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c258100-5235-4aee-b9a7-7e97267b0cb0-serviceca\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.386000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.384172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.389115 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.389083    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:50.389115 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.389114    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:50.389273 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.389128    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:50.389273 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.389195    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:50.889178503 +0000 UTC m=+3.043546983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:50.393766 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.393706    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjgm\" (UniqueName: \"kubernetes.io/projected/5fbf53cb-b32a-4e88-b5c1-2959df28a398-kube-api-access-8pjgm\") pod \"node-resolver-px5bd\" (UID: \"5fbf53cb-b32a-4e88-b5c1-2959df28a398\") " pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.394228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.394202    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlsn\" (UniqueName: \"kubernetes.io/projected/9ad82e4b-7d28-409e-a9b3-0bb88ba2a914-kube-api-access-6qlsn\") pod \"multus-additional-cni-plugins-ql8n6\" (UID: \"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914\") " pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.394482 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.394421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9dz\" (UniqueName: \"kubernetes.io/projected/7aff6fb0-49d5-4729-b743-fdf79154f869-kube-api-access-ks9dz\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.394482 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.394457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgr75\" (UniqueName: \"kubernetes.io/projected/067e10fb-5b75-446b-bc7c-7a7967b484ce-kube-api-access-wgr75\") pod \"aws-ebs-csi-driver-node-mwfh2\" (UID: \"067e10fb-5b75-446b-bc7c-7a7967b484ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.395430 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.395386    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglml\" (UniqueName: \"kubernetes.io/projected/9c258100-5235-4aee-b9a7-7e97267b0cb0-kube-api-access-zglml\") pod \"node-ca-khm77\" (UID: \"9c258100-5235-4aee-b9a7-7e97267b0cb0\") " pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.397259 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.397208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nqs2\" (UniqueName: \"kubernetes.io/projected/cfc2c1a6-b49e-406c-a839-ca526167e28a-kube-api-access-8nqs2\") pod \"multus-zjntw\" (UID: \"cfc2c1a6-b49e-406c-a839-ca526167e28a\") " pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.475795 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.475748    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hglmw"
Sep 15 18:50:50.488528 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.488321    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:50:50.489133 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:50.489099    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c7b98b_a7e4_44ce_ab7a_0cc1492d8b20.slice/crio-5d09bd415c3436d19aed6f9e2177ec6d12bd269ebb3273ffc8222d71f10ddc18 WatchSource:0}: Error finding container 5d09bd415c3436d19aed6f9e2177ec6d12bd269ebb3273ffc8222d71f10ddc18: Status 404 returned error can't find the container with id 5d09bd415c3436d19aed6f9e2177ec6d12bd269ebb3273ffc8222d71f10ddc18
Sep 15 18:50:50.499929 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.499855    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f5fws"
Sep 15 18:50:50.500577 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:50.500542    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364d3196_120d_4ad1_b2b5_7c5d71b0351f.slice/crio-ab996d23e9198ca136e42d287c47ada70a66e4d4ebb51af1a4a57037537ad104 WatchSource:0}: Error finding container ab996d23e9198ca136e42d287c47ada70a66e4d4ebb51af1a4a57037537ad104: Status 404 returned error can't find the container with id ab996d23e9198ca136e42d287c47ada70a66e4d4ebb51af1a4a57037537ad104
Sep 15 18:50:50.507579 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.507554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:50:50.510503 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:50.510476    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc798e072_df04_4f79_a74b_ba6102067c23.slice/crio-3a1b7cc5708f3b11486f5640eb12d4dde90991ee3bdfd99df39437e46ab3e8e4 WatchSource:0}: Error finding container 3a1b7cc5708f3b11486f5640eb12d4dde90991ee3bdfd99df39437e46ab3e8e4: Status 404 returned error can't find the container with id 3a1b7cc5708f3b11486f5640eb12d4dde90991ee3bdfd99df39437e46ab3e8e4
Sep 15 18:50:50.517884 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.517650    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-khm77"
Sep 15 18:50:50.518274 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:50.518247    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da97d55_3907_44e9_bb0c_b5d1c044bfb1.slice/crio-53d6ac357a0bdef2126da4d98ab931e0f4867219b0ac3c38644b68b4bda977e6 WatchSource:0}: Error finding container 53d6ac357a0bdef2126da4d98ab931e0f4867219b0ac3c38644b68b4bda977e6: Status 404 returned error can't find the container with id 53d6ac357a0bdef2126da4d98ab931e0f4867219b0ac3c38644b68b4bda977e6
Sep 15 18:50:50.525839 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.525816    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2"
Sep 15 18:50:50.534740 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.534705    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-px5bd"
Sep 15 18:50:50.543990 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.543964    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zjntw"
Sep 15 18:50:50.549968 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.549938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ql8n6"
Sep 15 18:50:50.843101 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.843019    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 18:50:50.887171 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.887134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:50.887330 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.887186    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:50.887330 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.887274    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:50.887410 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.887334    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:50.887410 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.887346    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:51.887329145 +0000 UTC m=+4.041697605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:50.887410 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.887380    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:51.887367701 +0000 UTC m=+4.041736148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:50.988403 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:50.988360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:50.988574 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.988521    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:50.988574 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.988548    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:50.988574 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.988562    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:50.988773 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:50.988646    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:51.988625443 +0000 UTC m=+4.142993903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:51.213363 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.213170    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:50 +0000 UTC" deadline="2028-02-11 01:50:50.315482147 +0000 UTC"
Sep 15 18:50:51.213363 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.213254    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12318h59m59.102233652s"
Sep 15 18:50:51.236360 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:51.236319    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c258100_5235_4aee_b9a7_7e97267b0cb0.slice/crio-cef8f7eb0d00722b1550b39270569ba0bddb9975f7b23d0651e31b00b999ab23 WatchSource:0}: Error finding container cef8f7eb0d00722b1550b39270569ba0bddb9975f7b23d0651e31b00b999ab23: Status 404 returned error can't find the container with id cef8f7eb0d00722b1550b39270569ba0bddb9975f7b23d0651e31b00b999ab23
Sep 15 18:50:51.242390 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.242351    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"53d6ac357a0bdef2126da4d98ab931e0f4867219b0ac3c38644b68b4bda977e6"}
Sep 15 18:50:51.243908 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.243872    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f5fws" event={"ID":"c798e072-df04-4f79-a74b-ba6102067c23","Type":"ContainerStarted","Data":"3a1b7cc5708f3b11486f5640eb12d4dde90991ee3bdfd99df39437e46ab3e8e4"}
Sep 15 18:50:51.245050 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.245025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7zbgr" event={"ID":"364d3196-120d-4ad1-b2b5-7c5d71b0351f","Type":"ContainerStarted","Data":"ab996d23e9198ca136e42d287c47ada70a66e4d4ebb51af1a4a57037537ad104"}
Sep 15 18:50:51.246284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.246254    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hglmw" event={"ID":"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20","Type":"ContainerStarted","Data":"5d09bd415c3436d19aed6f9e2177ec6d12bd269ebb3273ffc8222d71f10ddc18"}
Sep 15 18:50:51.247643 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.247597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-khm77" event={"ID":"9c258100-5235-4aee-b9a7-7e97267b0cb0","Type":"ContainerStarted","Data":"cef8f7eb0d00722b1550b39270569ba0bddb9975f7b23d0651e31b00b999ab23"}
Sep 15 18:50:51.742039 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:51.741982    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067e10fb_5b75_446b_bc7c_7a7967b484ce.slice/crio-2d2ad22cd0acb1e0683e534beaa5c3448dab865404f67ff2d21cd1021a7c6c35 WatchSource:0}: Error finding container 2d2ad22cd0acb1e0683e534beaa5c3448dab865404f67ff2d21cd1021a7c6c35: Status 404 returned error can't find the container with id 2d2ad22cd0acb1e0683e534beaa5c3448dab865404f67ff2d21cd1021a7c6c35
Sep 15 18:50:51.747589 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:50:51.747261    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad82e4b_7d28_409e_a9b3_0bb88ba2a914.slice/crio-913a022ca6a101c5799285c9096522983eb7a01c320928ddc41da33ad5213e38 WatchSource:0}: Error finding container 913a022ca6a101c5799285c9096522983eb7a01c320928ddc41da33ad5213e38: Status 404 returned error can't find the container with id 913a022ca6a101c5799285c9096522983eb7a01c320928ddc41da33ad5213e38
Sep 15 18:50:51.895812 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.895768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:51.895949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.895823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:51.896006 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.895986    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:51.896058 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.896046    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:53.896028404 +0000 UTC m=+6.050396869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:51.896218 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.896201    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:51.896267 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.896258    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:53.896244075 +0000 UTC m=+6.050612536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:51.996415 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:51.996322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:51.996661 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.996499    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:51.996661 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.996522    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:51.996661 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.996535    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:51.996661 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:51.996591    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:53.996574073 +0000 UTC m=+6.150942536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:52.214190 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.214145    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 18:45:50 +0000 UTC" deadline="2028-06-15 05:57:58.738772112 +0000 UTC"
Sep 15 18:50:52.214190 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.214187    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15323h7m6.524589065s"
Sep 15 18:50:52.229913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.229881    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:52.230060 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:52.230015    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:50:52.230457 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.230412    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:52.230562 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:52.230521    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:50:52.230644 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.230580    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:52.230697 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:52.230662    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:50:52.267916 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.267791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" event={"ID":"b7e01db16d59f3dba97f5f5f5aaef31d","Type":"ContainerStarted","Data":"9ab9c39f5547b91129dcccea3383168b5754a9b037cc6c768c7351fdda211ae2"}
Sep 15 18:50:52.272942 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.272857    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zjntw" event={"ID":"cfc2c1a6-b49e-406c-a839-ca526167e28a","Type":"ContainerStarted","Data":"827af864d247ba285928a8c3b6973f2a64f602356a7772c644125463992be6a8"}
Sep 15 18:50:52.277072 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.277004    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerStarted","Data":"913a022ca6a101c5799285c9096522983eb7a01c320928ddc41da33ad5213e38"}
Sep 15 18:50:52.279232 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.279168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-px5bd" event={"ID":"5fbf53cb-b32a-4e88-b5c1-2959df28a398","Type":"ContainerStarted","Data":"4ac78139512b513a6adcc93a08bd6efc769505610649bc6f8417790e3401917a"}
Sep 15 18:50:52.281579 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.281522    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-252.ec2.internal" podStartSLOduration=3.281504074 podStartE2EDuration="3.281504074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:50:52.280808582 +0000 UTC m=+4.435177078" watchObservedRunningTime="2026-09-15 18:50:52.281504074 +0000 UTC m=+4.435872543"
Sep 15 18:50:52.287552 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:52.287485    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2" event={"ID":"067e10fb-5b75-446b-bc7c-7a7967b484ce","Type":"ContainerStarted","Data":"2d2ad22cd0acb1e0683e534beaa5c3448dab865404f67ff2d21cd1021a7c6c35"}
Sep 15 18:50:53.313703 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:53.313299    2584 generic.go:356] "Generic (PLEG): container finished" podID="3003a6cfa64b8558b5eab03a3328f05d" containerID="2663a453f233667c518f90d7942b4875070e9e32e21e0288607480f0214e60aa" exitCode=0
Sep 15 18:50:53.316650 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:53.316140    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" event={"ID":"3003a6cfa64b8558b5eab03a3328f05d","Type":"ContainerDied","Data":"2663a453f233667c518f90d7942b4875070e9e32e21e0288607480f0214e60aa"}
Sep 15 18:50:53.913359 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:53.913320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:53.913550 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:53.913415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:53.913550 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:53.913531    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:53.913697 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:53.913594    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:57.913574523 +0000 UTC m=+10.067942991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:53.914051 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:53.914028    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:53.914156 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:53.914084    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:57.914069656 +0000 UTC m=+10.068438106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:54.013815 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:54.013772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:54.013989 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.013972    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:54.014075 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.013998    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:54.014075 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.014011    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:54.014075 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.014068    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:50:58.01405046 +0000 UTC m=+10.168418929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:54.229265 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:54.228718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:54.229265 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.228862    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:50:54.229265 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:54.228926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:54.229265 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.229003    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:50:54.229265 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:54.228716    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:54.229265 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:54.229075    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:50:56.228193 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:56.227356    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:56.228193 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:56.227500    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:50:56.228193 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:56.227903    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:56.228193 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:56.228013    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:50:56.228193 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:56.228066    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:56.228193 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:56.228138    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:50:57.951978 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:57.951936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:57.952560 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:57.951998    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:57.952560 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:57.952144    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:57.952560 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:57.952206    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:05.952186756 +0000 UTC m=+18.106555216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:50:57.952560 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:57.952262    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:57.952560 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:57.952302    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:05.952290361 +0000 UTC m=+18.106658816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:50:58.052879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:58.052809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:58.053060 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.053018    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:50:58.053060 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.053041    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:50:58.053060 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.053055    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:58.053218 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.053116    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:06.053095029 +0000 UTC m=+18.207463477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:50:58.232392 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:58.230758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:50:58.232392 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.230903    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:50:58.232392 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:58.231037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:50:58.232392 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:50:58.231064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:50:58.232392 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.231194    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:50:58.232392 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:50:58.231454    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:00.227144 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:00.227104    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:00.227144 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:00.227137    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:00.227648 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:00.227105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:00.227648 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:00.227249    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:00.227648 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:00.227326    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:00.227648 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:00.227427    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:02.227296 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:02.227258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:02.227296 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:02.227292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:02.227860 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:02.227258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:02.227860 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:02.227380    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:02.227860 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:02.227471    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:02.227860 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:02.227526    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:04.227865 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:04.227828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:04.228261 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:04.227828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:04.228261 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:04.227952    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:04.228261 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:04.228066    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:04.228261 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:04.227828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:04.228261 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:04.228194    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:06.011811 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:06.011765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:06.012267 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:06.011820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:06.012267 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.011930    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:06.012267 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.012002    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:06.012267 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.012016    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:22.011995211 +0000 UTC m=+34.166363672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:06.012267 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.012039    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:22.012028935 +0000 UTC m=+34.166397389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:06.112508 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:06.112463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:06.112702 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.112639    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:06.112702 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.112659    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:06.112702 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.112672    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:06.112820 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.112732    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:22.112714238 +0000 UTC m=+34.267082691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:06.227768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:06.227726    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:06.227949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:06.227734    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:06.227949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:06.227849    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:06.227949 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.227847    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:06.228101 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.227962    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:06.228101 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:06.228047    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:08.230690 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:08.230656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:08.230690 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:08.230673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:08.231186 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:08.230656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:08.231186 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:08.230769    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:08.231186 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:08.230864    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:08.231186 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:08.230975    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:09.348460 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.348213    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2" event={"ID":"067e10fb-5b75-446b-bc7c-7a7967b484ce","Type":"ContainerStarted","Data":"aad86c7a1e223bda463f91ec5fc210260fe44ef8ca8f07b0213fd6b94b45f93d"}
Sep 15 18:51:09.349799 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.349761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7zbgr" event={"ID":"364d3196-120d-4ad1-b2b5-7c5d71b0351f","Type":"ContainerStarted","Data":"8ac2af8497dea0f36117df7f248076c3fa0a7fa3b5773ec92bdd580c9b1f05cf"}
Sep 15 18:51:09.351290 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.351260    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hglmw" event={"ID":"39c7b98b-a7e4-44ce-ab7a-0cc1492d8b20","Type":"ContainerStarted","Data":"5074aa1025ea6400194146f34a99f520c163b695dc9bf3b099862b470d59d827"}
Sep 15 18:51:09.352784 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.352753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zjntw" event={"ID":"cfc2c1a6-b49e-406c-a839-ca526167e28a","Type":"ContainerStarted","Data":"fa88ef3cebd5b7fc8770facc817380b3a8e9b792f3e39d1877f943cee9f77b33"}
Sep 15 18:51:09.355057 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.355028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" event={"ID":"3003a6cfa64b8558b5eab03a3328f05d","Type":"ContainerStarted","Data":"88b82b9b10afa443837354b23c9d07712781a0d562cad4a899014791bb13dbe8"}
Sep 15 18:51:09.356660 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.356636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-khm77" event={"ID":"9c258100-5235-4aee-b9a7-7e97267b0cb0","Type":"ContainerStarted","Data":"43b913a3f9481b7632f398b5cb0552d52129b805775fdf4cc3290d8dfc5c7e5d"}
Sep 15 18:51:09.358365 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.358335    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ad82e4b-7d28-409e-a9b3-0bb88ba2a914" containerID="8d7e5f911b128618741515862e6a81d462eca931ca61bb5ce043780ab5c492e5" exitCode=0
Sep 15 18:51:09.358466 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.358408    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerDied","Data":"8d7e5f911b128618741515862e6a81d462eca931ca61bb5ce043780ab5c492e5"}
Sep 15 18:51:09.360295 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.360268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-px5bd" event={"ID":"5fbf53cb-b32a-4e88-b5c1-2959df28a398","Type":"ContainerStarted","Data":"26259ed16063513855ffc59f7bf6c06c051e26103b3110b5c5e35ddb0404f892"}
Sep 15 18:51:09.364085 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.363972    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7zbgr" podStartSLOduration=2.965872648 podStartE2EDuration="21.363960306s" totalImagesPullingTime="18.398087658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:09.363565648 +0000 UTC m=+21.517934116" watchObservedRunningTime="2026-09-15 18:51:09.363960306 +0000 UTC m=+21.518328773"
Sep 15 18:51:09.399357 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.399312    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-252.ec2.internal" podStartSLOduration=20.399297373 podStartE2EDuration="20.399297373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:09.399140439 +0000 UTC m=+21.553508908" watchObservedRunningTime="2026-09-15 18:51:09.399297373 +0000 UTC m=+21.553665841"
Sep 15 18:51:09.415405 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.415337    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hglmw" podStartSLOduration=2.976383644 podStartE2EDuration="21.415314798s" totalImagesPullingTime="18.438931154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:09.41389025 +0000 UTC m=+21.568258719" watchObservedRunningTime="2026-09-15 18:51:09.415314798 +0000 UTC m=+21.569683267"
Sep 15 18:51:09.428072 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.428019    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-khm77" podStartSLOduration=3.735745255 podStartE2EDuration="21.428002766s" totalImagesPullingTime="17.692257511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:09.427518214 +0000 UTC m=+21.581886683" watchObservedRunningTime="2026-09-15 18:51:09.428002766 +0000 UTC m=+21.582371233"
Sep 15 18:51:09.440217 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.440166    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-px5bd" podStartSLOduration=4.256776265 podStartE2EDuration="21.440150737s" totalImagesPullingTime="17.183374472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:09.44004146 +0000 UTC m=+21.594409929" watchObservedRunningTime="2026-09-15 18:51:09.440150737 +0000 UTC m=+21.594519206"
Sep 15 18:51:09.454967 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:09.454922    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zjntw" podStartSLOduration=4.228324437 podStartE2EDuration="21.454908104s" totalImagesPullingTime="17.226583667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:09.454736384 +0000 UTC m=+21.609104851" watchObservedRunningTime="2026-09-15 18:51:09.454908104 +0000 UTC m=+21.609276573"
Sep 15 18:51:10.227695 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.227040    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:10.227695 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.227040    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:10.227695 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:10.227158    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:10.227695 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:10.227250    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:10.227695 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.227040    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:10.227695 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:10.227338    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:10.364279 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.364238    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f5fws" event={"ID":"c798e072-df04-4f79-a74b-ba6102067c23","Type":"ContainerStarted","Data":"7802c3bf6aa7d595896a1b72747471e343b1b3347d9130760ab5fda5ab61f8d3"}
Sep 15 18:51:10.368734 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.368701    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"7402b69750cf4aae8e962f6bd3890bea66a92b75a095705561d4440ca8d3df89"}
Sep 15 18:51:10.368868 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.368795    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"ead3c958b115d06737336909f70005bd7442fcbd2c89c41d943d2e7b02d4358a"}
Sep 15 18:51:10.368868 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.368846    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"1fa1c53a062de0d5230c4fa7f43c146b67158a5499d5f06ddf5d6f02e33de205"}
Sep 15 18:51:10.368977 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.368917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"11e666a0c71bbbb2035c0303199033ec58b8fcb7a20eef17b535ec02c9b3efc3"}
Sep 15 18:51:10.368977 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.368967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"5885a9d2f639aa986ede16d7a2b95174fde2f21e1453e2b30af6b31eb8a44f79"}
Sep 15 18:51:10.369087 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.369014    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"1887c641583d47741f6dcb099d162d5cdc510a241636178c6348200ac6171e29"}
Sep 15 18:51:10.642695 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:10.642662    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 18:51:11.242360 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:11.242250    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T18:51:10.64268902Z","UUID":"88c727fb-633b-4215-8e61-69ebaf502d66","Handler":null,"Name":"","Endpoint":""}
Sep 15 18:51:11.244022 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:11.243988    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 18:51:11.244162 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:11.244028    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 18:51:11.373327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:11.373286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2" event={"ID":"067e10fb-5b75-446b-bc7c-7a7967b484ce","Type":"ContainerStarted","Data":"653b88846ecd7c9819a107e93187ac1e41e0a2da7c8888c102bfaab2d5fd806b"}
Sep 15 18:51:12.227912 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.227624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:12.227912 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.227624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:12.228211 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:12.227972    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:12.228211 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.227624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:12.228211 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:12.228030    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:12.228211 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:12.228095    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:12.377874 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.377833    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2" event={"ID":"067e10fb-5b75-446b-bc7c-7a7967b484ce","Type":"ContainerStarted","Data":"855e6c112af7bf7f19ec0b623f31477006c1957ed3be6210b30f2afd3291c003"}
Sep 15 18:51:12.381911 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.381887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"f166da1f867e72c505e96594476cfe57d8393bdc77aa03ae459a1ef7af282b38"}
Sep 15 18:51:12.394196 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.394157    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f5fws" podStartSLOduration=6.007462776 podStartE2EDuration="24.394144335s" totalImagesPullingTime="18.386681559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:10.380952284 +0000 UTC m=+22.535320754" watchObservedRunningTime="2026-09-15 18:51:12.394144335 +0000 UTC m=+24.548512811"
Sep 15 18:51:12.394677 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:12.394650    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mwfh2" podStartSLOduration=4.633971855 podStartE2EDuration="24.394642141s" totalImagesPullingTime="19.760670286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:12.393554022 +0000 UTC m=+24.547922502" watchObservedRunningTime="2026-09-15 18:51:12.394642141 +0000 UTC m=+24.549010605"
Sep 15 18:51:14.144381 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.144130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:51:14.145492 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.144753    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:51:14.228479 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.228452    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:14.228645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.228454    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:14.228645 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:14.228590    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:14.228801 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:14.228703    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:14.228801 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.228458    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:14.228885 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:14.228817    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:14.389394 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.389308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" event={"ID":"9da97d55-3907-44e9-bb0c-b5d1c044bfb1","Type":"ContainerStarted","Data":"3d5e6017b9270285074f27650f0f1ae53ee4cd131f08c0e75262e3466f2208bd"}
Sep 15 18:51:14.389577 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.389548    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:51:14.391485 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.391309    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ad82e4b-7d28-409e-a9b3-0bb88ba2a914" containerID="2e62fc3f608977c3adc16d6e21b834c422a78c4aa8f613260d5fb8b3746781e6" exitCode=0
Sep 15 18:51:14.391592 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.391402    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerDied","Data":"2e62fc3f608977c3adc16d6e21b834c422a78c4aa8f613260d5fb8b3746781e6"}
Sep 15 18:51:14.391728 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.391710    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:51:14.392210 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.392138    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7zbgr"
Sep 15 18:51:14.405152 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.405130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:51:14.413084 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:14.413043    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx" podStartSLOduration=7.679418485 podStartE2EDuration="26.413030345s" totalImagesPullingTime="18.73361186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:14.411379081 +0000 UTC m=+26.565747549" watchObservedRunningTime="2026-09-15 18:51:14.413030345 +0000 UTC m=+26.567398813"
Sep 15 18:51:15.395140 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.395103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:51:15.395679 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.395659    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:51:15.412421 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.412188    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:51:15.982777 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.982745    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nz4mf"]
Sep 15 18:51:15.982941 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.982871    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:15.982997 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:15.982974    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:15.983476 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.983446    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x77j7"]
Sep 15 18:51:15.983604 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.983552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:15.983670 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:15.983658    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:15.984799 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.984775    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqw9v"]
Sep 15 18:51:15.984896 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:15.984870    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:15.984968 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:15.984946    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:16.398432 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:16.398402    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ad82e4b-7d28-409e-a9b3-0bb88ba2a914" containerID="b0497cf97686b8897f5e904f1c120a388452d50f06fb34464fe73701fcceb1ee" exitCode=0
Sep 15 18:51:16.399305 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:16.398449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerDied","Data":"b0497cf97686b8897f5e904f1c120a388452d50f06fb34464fe73701fcceb1ee"}
Sep 15 18:51:17.227930 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:17.227635    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:17.228216 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:17.228072    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:17.402935 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:17.402896    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ad82e4b-7d28-409e-a9b3-0bb88ba2a914" containerID="ef61bb9516b31f415d2067bfad46613c1e9726752ccefda01f84a0e3c3ac3aff" exitCode=0
Sep 15 18:51:17.403313 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:17.402959    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerDied","Data":"ef61bb9516b31f415d2067bfad46613c1e9726752ccefda01f84a0e3c3ac3aff"}
Sep 15 18:51:18.229014 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:18.228557    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:18.229014 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:18.228685    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:18.229014 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:18.228745    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:18.229014 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:18.228879    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:19.226998 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:19.226965    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:19.227460 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:19.227097    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:20.227578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:20.227492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:20.228119 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:20.227649    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nz4mf" podUID="ac6cee9d-5017-44fb-b69c-d90c689c7cf2"
Sep 15 18:51:20.228119 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:20.227706    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:20.228119 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:20.227837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:51:20.366830 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:20.366779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:20.454493 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:20.454455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:20.559089 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:20.559054    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:21.039762 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.039733    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:21.093650 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.093599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:21.227626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.227594    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:21.228191 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:21.227715    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jqw9v" podUID="53c11c0f-bbb6-49d8-896e-d17c292a1644"
Sep 15 18:51:21.327556 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.327478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:21.389653 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.389605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:21.717083 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.717054    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-252.ec2.internal" event="NodeReady"
Sep 15 18:51:21.717253 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.717186    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 18:51:21.748289 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.748261    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-665c487469-v2k97"]
Sep 15 18:51:21.761556 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.761529    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-58p5d"]
Sep 15 18:51:21.761685 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.761670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.764627 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.764586    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 18:51:21.764893 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.764875    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwrs6\""
Sep 15 18:51:21.765000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.764901    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 18:51:21.765000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.764914    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 18:51:21.775330 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.775291    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 18:51:21.775639 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.775584    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7jnzj"]
Sep 15 18:51:21.775767 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.775740    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:21.777909 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.777886    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 18:51:21.778026 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.777961    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hj774\""
Sep 15 18:51:21.778026 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.777888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 18:51:21.796596 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.796565    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-665c487469-v2k97"]
Sep 15 18:51:21.796596 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.796595    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-58p5d"]
Sep 15 18:51:21.796756 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.796634    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7jnzj"]
Sep 15 18:51:21.796756 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.796698    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:21.799102 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.799085    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 18:51:21.799102 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.799096    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xmcwz\""
Sep 15 18:51:21.799276 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.799182    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 18:51:21.799276 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.799189    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 18:51:21.929207 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929135    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c098a940-98ba-4e64-a691-e2cd0d6a41bc-config-volume\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:21.929207 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swms9\" (UniqueName: \"kubernetes.io/projected/8a47024c-545b-4dc3-b464-77bc724c4d1d-kube-api-access-swms9\") pod \"ingress-canary-7jnzj\" (UID: \"8a47024c-545b-4dc3-b464-77bc724c4d1d\") " pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:21.929369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929246    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5jw\" (UniqueName: \"kubernetes.io/projected/c098a940-98ba-4e64-a691-e2cd0d6a41bc-kube-api-access-4h5jw\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:21.929369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-bound-sa-token\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929312    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a47024c-545b-4dc3-b464-77bc724c4d1d-cert\") pod \"ingress-canary-7jnzj\" (UID: \"8a47024c-545b-4dc3-b464-77bc724c4d1d\") " pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:21.929369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:21.929369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-image-registry-private-configuration\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-certificates\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gv4g\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-kube-api-access-9gv4g\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c098a940-98ba-4e64-a691-e2cd0d6a41bc-tmp-dir\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:21.929570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929435    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-installation-pull-secrets\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18c248e9-c09c-48da-b1bf-9ec0401841f5-ca-trust-extracted\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:21.929570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:21.929529    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-trusted-ca\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.030745 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.030720    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:22.030873 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.030779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c098a940-98ba-4e64-a691-e2cd0d6a41bc-config-volume\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.030873 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.030801    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swms9\" (UniqueName: \"kubernetes.io/projected/8a47024c-545b-4dc3-b464-77bc724c4d1d-kube-api-access-swms9\") pod \"ingress-canary-7jnzj\" (UID: \"8a47024c-545b-4dc3-b464-77bc724c4d1d\") " pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:22.030873 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.030821    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5jw\" (UniqueName: \"kubernetes.io/projected/c098a940-98ba-4e64-a691-e2cd0d6a41bc-kube-api-access-4h5jw\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.030873 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.030844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.030873 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.030863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-bound-sa-token\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.030873 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.030866    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.030930    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.03091052 +0000 UTC m=+66.185278980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031004    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031021    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031037    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a47024c-545b-4dc3-b464-77bc724c4d1d-cert\") pod \"ingress-canary-7jnzj\" (UID: \"8a47024c-545b-4dc3-b464-77bc724c4d1d\") " pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031072    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:22.531054884 +0000 UTC m=+34.685423331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031094    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031128    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-image-registry-private-configuration\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031157 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-certificates\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031174    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gv4g\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-kube-api-access-9gv4g\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031227    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:51:22.531211183 +0000 UTC m=+34.685579649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031345    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c098a940-98ba-4e64-a691-e2cd0d6a41bc-tmp-dir\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-installation-pull-secrets\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18c248e9-c09c-48da-b1bf-9ec0401841f5-ca-trust-extracted\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-trusted-ca\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031904 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031735    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c098a940-98ba-4e64-a691-e2cd0d6a41bc-tmp-dir\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.031904 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c098a940-98ba-4e64-a691-e2cd0d6a41bc-config-volume\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.031904 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.031858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-certificates\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.031904 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031870    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:22.032075 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.031926    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret podName:53c11c0f-bbb6-49d8-896e-d17c292a1644 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.031907812 +0000 UTC m=+66.186276272 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret") pod "global-pull-secret-syncer-jqw9v" (UID: "53c11c0f-bbb6-49d8-896e-d17c292a1644") : object "kube-system"/"original-pull-secret" not registered
Sep 15 18:51:22.032189 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.032162    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18c248e9-c09c-48da-b1bf-9ec0401841f5-ca-trust-extracted\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.032790 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.032703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-trusted-ca\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.035574 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.035553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-image-registry-private-configuration\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.035681 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.035577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a47024c-545b-4dc3-b464-77bc724c4d1d-cert\") pod \"ingress-canary-7jnzj\" (UID: \"8a47024c-545b-4dc3-b464-77bc724c4d1d\") " pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:22.035727 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.035691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-installation-pull-secrets\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.038676 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.038654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5jw\" (UniqueName: \"kubernetes.io/projected/c098a940-98ba-4e64-a691-e2cd0d6a41bc-kube-api-access-4h5jw\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.038763 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.038691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gv4g\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-kube-api-access-9gv4g\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.039001 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.038985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swms9\" (UniqueName: \"kubernetes.io/projected/8a47024c-545b-4dc3-b464-77bc724c4d1d-kube-api-access-swms9\") pod \"ingress-canary-7jnzj\" (UID: \"8a47024c-545b-4dc3-b464-77bc724c4d1d\") " pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:22.039557 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.039540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-bound-sa-token\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.104554 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.104411    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7jnzj"
Sep 15 18:51:22.131843 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.131818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:22.132004 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.131974    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 18:51:22.132004 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.131992    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 18:51:22.132004 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.132001    2584 projected.go:196] Error preparing data for projected volume kube-api-access-kb58t for pod openshift-network-diagnostics/network-check-target-nz4mf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:22.132106 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.132058    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t podName:ac6cee9d-5017-44fb-b69c-d90c689c7cf2 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:54.132042794 +0000 UTC m=+66.286411259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kb58t" (UniqueName: "kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t") pod "network-check-target-nz4mf" (UID: "ac6cee9d-5017-44fb-b69c-d90c689c7cf2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 18:51:22.230209 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.230183    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:22.230686 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.230358    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:22.232624 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.232560    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:22.232624 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.232593    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:22.232782 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.232727    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:22.232782 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.232755    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bc8c\""
Sep 15 18:51:22.232878 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.232847    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrmrg\""
Sep 15 18:51:22.417225 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.417187    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ad82e4b-7d28-409e-a9b3-0bb88ba2a914" containerID="bdc60f7b78d680a22154964819e1fcd2eab7b87176904d552f198a6a13527c5c" exitCode=0
Sep 15 18:51:22.417366 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.417246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerDied","Data":"bdc60f7b78d680a22154964819e1fcd2eab7b87176904d552f198a6a13527c5c"}
Sep 15 18:51:22.429559 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.429534    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7jnzj"]
Sep 15 18:51:22.433822 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:51:22.433797    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a47024c_545b_4dc3_b464_77bc724c4d1d.slice/crio-fe4707747655b707db2dd39467a7b4ed7a570b8dea5dab08c0baccc7a6a2dbf6 WatchSource:0}: Error finding container fe4707747655b707db2dd39467a7b4ed7a570b8dea5dab08c0baccc7a6a2dbf6: Status 404 returned error can't find the container with id fe4707747655b707db2dd39467a7b4ed7a570b8dea5dab08c0baccc7a6a2dbf6
Sep 15 18:51:22.534262 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.534233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:22.534396 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:22.534304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:22.534396 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.534385    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:22.534568 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.534405    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:22.534568 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.534411    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:22.534568 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.534464    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.534443404 +0000 UTC m=+35.688811863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:22.534568 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:22.534483    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:51:23.534472538 +0000 UTC m=+35.688840983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:23.227818 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.227782    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:23.230368 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.230345    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:23.421511 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.421466    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7jnzj" event={"ID":"8a47024c-545b-4dc3-b464-77bc724c4d1d","Type":"ContainerStarted","Data":"fe4707747655b707db2dd39467a7b4ed7a570b8dea5dab08c0baccc7a6a2dbf6"}
Sep 15 18:51:23.425142 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.425104    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ad82e4b-7d28-409e-a9b3-0bb88ba2a914" containerID="9758079b78ae2b15331948e95082da0496860681ea80aff8bd0bd00fed9521b0" exitCode=0
Sep 15 18:51:23.425290 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.425142    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerDied","Data":"9758079b78ae2b15331948e95082da0496860681ea80aff8bd0bd00fed9521b0"}
Sep 15 18:51:23.542654 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.542628    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:23.542794 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:23.542689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:23.542857 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:23.542830    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:23.542857 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:23.542851    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:23.542951 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:23.542911    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:25.542888227 +0000 UTC m=+37.697256689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:23.542951 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:23.542928    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:23.543077 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:23.542978    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:51:25.542968619 +0000 UTC m=+37.697337067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:24.431901 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:24.431855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" event={"ID":"9ad82e4b-7d28-409e-a9b3-0bb88ba2a914","Type":"ContainerStarted","Data":"81c221df52f60a20d8d112de75cc7d297aa5a80828a5ebaf6d0ac24a8671e1d8"}
Sep 15 18:51:24.459038 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:24.458982    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ql8n6" podStartSLOduration=8.695586582 podStartE2EDuration="36.458963585s" totalImagesPullingTime="27.763377003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:24.457601661 +0000 UTC m=+36.611970131" watchObservedRunningTime="2026-09-15 18:51:24.458963585 +0000 UTC m=+36.613332053"
Sep 15 18:51:24.584365 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:24.584326    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:24.685303 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:24.685196    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:25.557773 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:25.557506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:25.558165 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:25.557655    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:25.558165 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:25.557887    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:51:29.557872011 +0000 UTC m=+41.712240457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:25.558165 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:25.557936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:25.558165 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:25.558079    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:25.558165 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:25.558097    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:25.558165 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:25.558159    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:29.558140863 +0000 UTC m=+41.712509329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:26.438053 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:26.437960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7jnzj" event={"ID":"8a47024c-545b-4dc3-b464-77bc724c4d1d","Type":"ContainerStarted","Data":"4e6994d5851404894bee4fa9ac5eb1b7ed850724bcd62baa04d46bb21d3daa67"}
Sep 15 18:51:26.454438 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:26.454391    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7jnzj" podStartSLOduration=2.514861737 podStartE2EDuration="5.45437882s" totalImagesPullingTime="2.939517083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:26.452851514 +0000 UTC m=+38.607219994" watchObservedRunningTime="2026-09-15 18:51:26.45437882 +0000 UTC m=+38.608747289"
Sep 15 18:51:27.409448 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:27.409419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:28.479086 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:28.479055    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:28.580204 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:28.580176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:28.908739 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:28.908708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:28.963560 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:28.963532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:29.585097 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:29.585064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:29.585453 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:29.585106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:29.585453 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:29.585227    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:29.585453 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:29.585246    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:29.585453 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:29.585295    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:37.585281342 +0000 UTC m=+49.739649793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:29.585453 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:29.585225    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:29.585453 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:29.585342    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:51:37.585331225 +0000 UTC m=+49.739699671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:37.645098 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:37.645055    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:37.645543 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:37.645139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:37.645543 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:37.645221    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:37.645543 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:37.645237    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:37.645543 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:37.645282    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:51:53.645268963 +0000 UTC m=+65.799637408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:37.645543 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:37.645220    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:37.645543 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:37.645376    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:51:53.645360807 +0000 UTC m=+65.799729254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:43.037107 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:43.037070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:43.091548 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:43.091525    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:47.413238 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:47.413216    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qvxx"
Sep 15 18:51:47.561909 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:47.561876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:47.624360 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:47.624329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:50.487090 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:50.487053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:50.588437 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:50.588407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:53.662522 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:53.662467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:51:53.662978 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:53.662536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:51:53.662978 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:53.662636    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:51:53.662978 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:53.662660    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:51:53.662978 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:53.662659    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:51:53.662978 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:53.662715    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:25.662698503 +0000 UTC m=+97.817066968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:51:53.662978 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:53.662728    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:52:25.662722081 +0000 UTC m=+97.817090527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:51:54.065146 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.065107    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:51:54.065333 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.065308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:54.068009 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.067990    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 18:51:54.068054 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.067991    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 18:51:54.076119 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:54.076100    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:51:54.076164 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:51:54.076157    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:52:58.076137356 +0000 UTC m=+130.230505802 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : secret "metrics-daemon-secret" not found
Sep 15 18:51:54.079521 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.079496    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/53c11c0f-bbb6-49d8-896e-d17c292a1644-original-pull-secret\") pod \"global-pull-secret-syncer-jqw9v\" (UID: \"53c11c0f-bbb6-49d8-896e-d17c292a1644\") " pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:54.136070 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.136042    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jqw9v"
Sep 15 18:51:54.166274 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.166242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:54.168965 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.168941    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 18:51:54.178563 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.178538    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 18:51:54.189053 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.189029    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb58t\" (UniqueName: \"kubernetes.io/projected/ac6cee9d-5017-44fb-b69c-d90c689c7cf2-kube-api-access-kb58t\") pod \"network-check-target-nz4mf\" (UID: \"ac6cee9d-5017-44fb-b69c-d90c689c7cf2\") " pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:54.342064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.341987    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mrmrg\""
Sep 15 18:51:54.349824 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.349809    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:54.403724 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.403632    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jqw9v"]
Sep 15 18:51:54.407307 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:51:54.407273    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c11c0f_bbb6_49d8_896e_d17c292a1644.slice/crio-3a6ab83a0efb9a12a5533abfab5cece554edb4cd6ec1c33a124c2ddc1a76c588 WatchSource:0}: Error finding container 3a6ab83a0efb9a12a5533abfab5cece554edb4cd6ec1c33a124c2ddc1a76c588: Status 404 returned error can't find the container with id 3a6ab83a0efb9a12a5533abfab5cece554edb4cd6ec1c33a124c2ddc1a76c588
Sep 15 18:51:54.502507 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.502468    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqw9v" event={"ID":"53c11c0f-bbb6-49d8-896e-d17c292a1644","Type":"ContainerStarted","Data":"3a6ab83a0efb9a12a5533abfab5cece554edb4cd6ec1c33a124c2ddc1a76c588"}
Sep 15 18:51:54.616177 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:54.616103    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nz4mf"]
Sep 15 18:51:54.619340 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:51:54.619312    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6cee9d_5017_44fb_b69c_d90c689c7cf2.slice/crio-eef7c5acb309f34659f0c421a86f3ec64e8c5a9dd056f0b4b03ddf05c942f07c WatchSource:0}: Error finding container eef7c5acb309f34659f0c421a86f3ec64e8c5a9dd056f0b4b03ddf05c942f07c: Status 404 returned error can't find the container with id eef7c5acb309f34659f0c421a86f3ec64e8c5a9dd056f0b4b03ddf05c942f07c
Sep 15 18:51:55.507675 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:55.507634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nz4mf" event={"ID":"ac6cee9d-5017-44fb-b69c-d90c689c7cf2","Type":"ContainerStarted","Data":"eef7c5acb309f34659f0c421a86f3ec64e8c5a9dd056f0b4b03ddf05c942f07c"}
Sep 15 18:51:56.193275 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:56.193235    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:56.255771 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:56.255707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:51:59.520106 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:59.520072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jqw9v" event={"ID":"53c11c0f-bbb6-49d8-896e-d17c292a1644","Type":"ContainerStarted","Data":"5a2ef989d0429b1596029cf0f183806514f835fd3d243ebbb88cec9f3274d3e9"}
Sep 15 18:51:59.521574 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:59.521551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nz4mf" event={"ID":"ac6cee9d-5017-44fb-b69c-d90c689c7cf2","Type":"ContainerStarted","Data":"af64aa6cdfd491071192767b741ce4ba48e243599635a947b10e23836a139956"}
Sep 15 18:51:59.521695 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:59.521659    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:51:59.534014 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:51:59.533967    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jqw9v" podStartSLOduration=66.979265772 podStartE2EDuration="1m11.533925763s" totalImagesPullingTime="4.554659991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:59.533210264 +0000 UTC m=+71.687578731" watchObservedRunningTime="2026-09-15 18:51:59.533925763 +0000 UTC m=+71.688294231"
Sep 15 18:52:19.933683 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:19.933646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:52:20.035300 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:20.035267    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:52:25.688819 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:25.688780    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:52:25.688819 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:25.688828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:52:25.689275 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:25.688928    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:52:25.689275 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:25.688946    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:52:25.689275 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:25.689005    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:53:29.688991336 +0000 UTC m=+161.843359783 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:52:25.689275 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:25.688948    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:52:25.689275 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:25.689078    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:53:29.689065899 +0000 UTC m=+161.843434345 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:52:30.525435 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:30.525406    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nz4mf"
Sep 15 18:52:30.540410 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:30.540364    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nz4mf" podStartSLOduration=98.202657133 podStartE2EDuration="1m42.540350194s" totalImagesPullingTime="4.337693061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:51:59.549230053 +0000 UTC m=+71.703598521" watchObservedRunningTime="2026-09-15 18:52:30.540350194 +0000 UTC m=+102.694718662"
Sep 15 18:52:34.213471 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.213442    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"]
Sep 15 18:52:34.216140 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.216121    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.219471 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.219449    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 18:52:34.219471 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.219467    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 18:52:34.219653 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.219488    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 18:52:34.219653 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.219583    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 18:52:34.225342 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.225323    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"]
Sep 15 18:52:34.348572 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.348537    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-klusterlet-config\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.348809 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.348672    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-tmp\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.348809 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.348700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzg2t\" (UniqueName: \"kubernetes.io/projected/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-kube-api-access-qzg2t\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.449300 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.449257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-tmp\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.449511 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.449310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzg2t\" (UniqueName: \"kubernetes.io/projected/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-kube-api-access-qzg2t\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.449511 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.449354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-klusterlet-config\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.449705 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.449682    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-tmp\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.451890 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.451868    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-klusterlet-config\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.457877 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.457858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzg2t\" (UniqueName: \"kubernetes.io/projected/71891a88-8906-43ce-9c1e-ce7acdc6dfe1-kube-api-access-qzg2t\") pod \"klusterlet-addon-workmgr-88fdf9bc8-j8xfd\" (UID: \"71891a88-8906-43ce-9c1e-ce7acdc6dfe1\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.526208 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.526159    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:34.790558 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:34.790488    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"]
Sep 15 18:52:34.793327 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:52:34.793296    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71891a88_8906_43ce_9c1e_ce7acdc6dfe1.slice/crio-3a4374a5fa34e9ee7eb9d797c7b8e166dd4ec3eca8004f68f9e11ef2df103840 WatchSource:0}: Error finding container 3a4374a5fa34e9ee7eb9d797c7b8e166dd4ec3eca8004f68f9e11ef2df103840: Status 404 returned error can't find the container with id 3a4374a5fa34e9ee7eb9d797c7b8e166dd4ec3eca8004f68f9e11ef2df103840
Sep 15 18:52:35.611281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:35.611153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd" event={"ID":"71891a88-8906-43ce-9c1e-ce7acdc6dfe1","Type":"ContainerStarted","Data":"3a4374a5fa34e9ee7eb9d797c7b8e166dd4ec3eca8004f68f9e11ef2df103840"}
Sep 15 18:52:38.620901 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:38.620861    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd" event={"ID":"71891a88-8906-43ce-9c1e-ce7acdc6dfe1","Type":"ContainerStarted","Data":"17e10abefa3c08f28c78bbaf52083fce3b5854519e1429d1901985b01b1d937b"}
Sep 15 18:52:38.621274 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:38.621073    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:38.622642 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:38.622606    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:52:38.636148 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:38.636099    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd" podStartSLOduration=1.190370376 podStartE2EDuration="4.636084416s" totalImagesPullingTime="3.44571404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:52:38.63543468 +0000 UTC m=+110.789803148" watchObservedRunningTime="2026-09-15 18:52:38.636084416 +0000 UTC m=+110.790452884"
Sep 15 18:52:58.118333 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:52:58.118281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:52:58.118950 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:58.118442    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:52:58.118950 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:52:58.118523    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:00.118502341 +0000 UTC m=+252.272870807 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : secret "metrics-daemon-secret" not found
Sep 15 18:53:24.775258 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:24.775219    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-665c487469-v2k97" podUID="18c248e9-c09c-48da-b1bf-9ec0401841f5"
Sep 15 18:53:24.784376 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:24.784345    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-58p5d" podUID="c098a940-98ba-4e64-a691-e2cd0d6a41bc"
Sep 15 18:53:25.298778 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:25.298740    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:53:25.736770 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:53:25.736695    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:53:25.736770 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:53:25.736730    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-58p5d"
Sep 15 18:53:29.738849 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:53:29.738809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") pod \"image-registry-665c487469-v2k97\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") " pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:53:29.738849 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:53:29.738852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:53:29.739236 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:29.738955    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 18:53:29.739236 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:29.738960    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 18:53:29.739236 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:29.738978    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-665c487469-v2k97: secret "image-registry-tls" not found
Sep 15 18:53:29.739236 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:29.739014    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls podName:c098a940-98ba-4e64-a691-e2cd0d6a41bc nodeName:}" failed. No retries permitted until 2026-09-15 18:55:31.739000173 +0000 UTC m=+283.893368619 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls") pod "dns-default-58p5d" (UID: "c098a940-98ba-4e64-a691-e2cd0d6a41bc") : secret "dns-default-metrics-tls" not found
Sep 15 18:53:29.739236 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:53:29.739034    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls podName:18c248e9-c09c-48da-b1bf-9ec0401841f5 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:31.739019409 +0000 UTC m=+283.893387868 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls") pod "image-registry-665c487469-v2k97" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5") : secret "image-registry-tls" not found
Sep 15 18:53:39.227148 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:53:39.227105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:54:38.622317 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:38.622212    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd" podUID="71891a88-8906-43ce-9c1e-ce7acdc6dfe1" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Sep 15 18:54:38.909381 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:38.909296    2584 generic.go:356] "Generic (PLEG): container finished" podID="71891a88-8906-43ce-9c1e-ce7acdc6dfe1" containerID="17e10abefa3c08f28c78bbaf52083fce3b5854519e1429d1901985b01b1d937b" exitCode=1
Sep 15 18:54:38.909381 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:38.909344    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd" event={"ID":"71891a88-8906-43ce-9c1e-ce7acdc6dfe1","Type":"ContainerDied","Data":"17e10abefa3c08f28c78bbaf52083fce3b5854519e1429d1901985b01b1d937b"}
Sep 15 18:54:38.909731 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:38.909715    2584 scope.go:117] "RemoveContainer" containerID="17e10abefa3c08f28c78bbaf52083fce3b5854519e1429d1901985b01b1d937b"
Sep 15 18:54:39.913130 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:39.913095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd" event={"ID":"71891a88-8906-43ce-9c1e-ce7acdc6dfe1","Type":"ContainerStarted","Data":"b8b2c8563bdb08a1608bf868cfbcd0a83662662bf42f94de2934de528c6f4e9f"}
Sep 15 18:54:39.913539 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:39.913338    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:54:39.913916 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:39.913900    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-88fdf9bc8-j8xfd"
Sep 15 18:54:52.565313 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.565281    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-647fd6f876-2xxs8"]
Sep 15 18:54:52.567996 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.567981    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.570489 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.570461    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.570626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.570486    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 18:54:52.570626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.570468    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 18:54:52.570626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.570517    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 18:54:52.570626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.570557    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.570936 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.570916    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 18:54:52.571296 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.571281    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mpxn4\""
Sep 15 18:54:52.579840 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.579818    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-647fd6f876-2xxs8"]
Sep 15 18:54:52.665279 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.665251    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"]
Sep 15 18:54:52.667927 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.667912    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.670423 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670396    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 18:54:52.670544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670457    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:54:52.670544 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670484    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 18:54:52.670665 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670599    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 18:54:52.670718 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670695    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-default-certificate\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.670762 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.670830 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvbz\" (UniqueName: \"kubernetes.io/projected/042f4fd4-4470-4b60-927d-37c53e0f25d7-kube-api-access-wsvbz\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.670872 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670859    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-stats-auth\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.670923 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.670908    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.671078 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.671067    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zhnn5\""
Sep 15 18:54:52.676827 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.676804    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"]
Sep 15 18:54:52.723246 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.723224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:52.771767 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-default-certificate\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.771767 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.771913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de912687-0dfa-4dd1-8562-1e01213aee3f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.771913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97p8\" (UniqueName: \"kubernetes.io/projected/de912687-0dfa-4dd1-8562-1e01213aee3f-kube-api-access-b97p8\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.771913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvbz\" (UniqueName: \"kubernetes.io/projected/042f4fd4-4470-4b60-927d-37c53e0f25d7-kube-api-access-wsvbz\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.772032 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:52.771920    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 18:54:52.772032 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771930    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-stats-auth\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.772032 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.771972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.772032 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:52.771982    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:53.271966454 +0000 UTC m=+245.426334899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : secret "router-metrics-certs-default" not found
Sep 15 18:54:52.772032 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.772009    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de912687-0dfa-4dd1-8562-1e01213aee3f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.772299 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:52.772075    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:53.272060549 +0000 UTC m=+245.426429017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : configmap references non-existent config key: service-ca.crt
Sep 15 18:54:52.772901 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.772878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:52.774360 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.774339    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-default-certificate\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.774462 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.774449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-stats-auth\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.779237 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.779220    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvbz\" (UniqueName: \"kubernetes.io/projected/042f4fd4-4470-4b60-927d-37c53e0f25d7-kube-api-access-wsvbz\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:52.873209 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.873158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de912687-0dfa-4dd1-8562-1e01213aee3f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.873209 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.873184    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97p8\" (UniqueName: \"kubernetes.io/projected/de912687-0dfa-4dd1-8562-1e01213aee3f-kube-api-access-b97p8\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.873319 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.873227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de912687-0dfa-4dd1-8562-1e01213aee3f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.873619 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.873593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de912687-0dfa-4dd1-8562-1e01213aee3f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.875131 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.875111    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de912687-0dfa-4dd1-8562-1e01213aee3f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.880673 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.880651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97p8\" (UniqueName: \"kubernetes.io/projected/de912687-0dfa-4dd1-8562-1e01213aee3f-kube-api-access-b97p8\") pod \"kube-storage-version-migrator-operator-6c8f84f897-x9rlg\" (UID: \"de912687-0dfa-4dd1-8562-1e01213aee3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:52.976872 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:52.976851    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"
Sep 15 18:54:53.238334 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:53.238265    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg"]
Sep 15 18:54:53.241119 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:54:53.241092    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde912687_0dfa_4dd1_8562_1e01213aee3f.slice/crio-a439058f14502d69dcee3b4250bc1f6633db8c2f25a7dd908c9fa58fe5424a63 WatchSource:0}: Error finding container a439058f14502d69dcee3b4250bc1f6633db8c2f25a7dd908c9fa58fe5424a63: Status 404 returned error can't find the container with id a439058f14502d69dcee3b4250bc1f6633db8c2f25a7dd908c9fa58fe5424a63
Sep 15 18:54:53.275813 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:53.275793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:53.275904 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:53.275849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:53.275943 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:53.275932    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 18:54:53.275976 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:53.275946    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:54.27593062 +0000 UTC m=+246.430299065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : configmap references non-existent config key: service-ca.crt
Sep 15 18:54:53.275976 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:53.275971    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:54.275958717 +0000 UTC m=+246.430327164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : secret "router-metrics-certs-default" not found
Sep 15 18:54:53.948262 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:53.948225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg" event={"ID":"de912687-0dfa-4dd1-8562-1e01213aee3f","Type":"ContainerStarted","Data":"a439058f14502d69dcee3b4250bc1f6633db8c2f25a7dd908c9fa58fe5424a63"}
Sep 15 18:54:54.283247 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:54.283207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:54.283411 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:54.283363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:54.283411 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:54.283381    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 18:54:54.283494 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:54.283460    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:56.283437885 +0000 UTC m=+248.437806334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : secret "router-metrics-certs-default" not found
Sep 15 18:54:54.283494 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:54.283489    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:56.283471197 +0000 UTC m=+248.437839642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : configmap references non-existent config key: service-ca.crt
Sep 15 18:54:55.505244 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:55.505215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:55.956602 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:55.956520    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg" event={"ID":"de912687-0dfa-4dd1-8562-1e01213aee3f","Type":"ContainerStarted","Data":"5b059a79cebfe9b8b457ad5c1bda5720e40afd51b8e1e32c08149a54b8c1e821"}
Sep 15 18:54:55.970483 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:55.970435    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg" podStartSLOduration=1.676382048 podStartE2EDuration="3.970416765s" totalImagesPullingTime="2.294034717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:54:55.96960025 +0000 UTC m=+248.123968720" watchObservedRunningTime="2026-09-15 18:54:55.970416765 +0000 UTC m=+248.124785234"
Sep 15 18:54:56.300192 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:56.300155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:56.300348 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:56.300263    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:54:56.300348 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:56.300340    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:00.30031975 +0000 UTC m=+252.454688209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : configmap references non-existent config key: service-ca.crt
Sep 15 18:54:56.300442 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:56.300379    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 18:54:56.300442 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:56.300427    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:00.300415657 +0000 UTC m=+252.454784108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : secret "router-metrics-certs-default" not found
Sep 15 18:54:56.898657 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:56.898606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:56.998808 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:56.998775    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:57.931728 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:57.931696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:57.990690 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:57.990662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:58.184803 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.184729    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v6g27"]
Sep 15 18:54:58.187789 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.187774    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.190472 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.190450    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 18:54:58.191380 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.191363    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 18:54:58.191492 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.191378    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvqm8\""
Sep 15 18:54:58.191492 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.191362    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 18:54:58.191492 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.191370    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 18:54:58.198769 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.198745    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v6g27"]
Sep 15 18:54:58.201813 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.201793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:58.254367 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.254348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:58.313489 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.313471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-data-volume\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.313582 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.313510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-crio-socket\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.313582 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.313529    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.313684 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.313587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtmt\" (UniqueName: \"kubernetes.io/projected/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-api-access-zhtmt\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.313719 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.313684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.398531 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.398504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:58.414579 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414554    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-data-volume\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.414663 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414598    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-crio-socket\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.414663 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414635    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.414767 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtmt\" (UniqueName: \"kubernetes.io/projected/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-api-access-zhtmt\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.414767 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414695    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-crio-socket\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.414767 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.414925 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.414790    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.414925 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.414853    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:58.914838656 +0000 UTC m=+251.069207107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.414925 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.414789    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.415049 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.414938    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:58.914923599 +0000 UTC m=+251.069292050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.415049 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.414944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-data-volume\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.425060 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.425038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtmt\" (UniqueName: \"kubernetes.io/projected/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-api-access-zhtmt\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.462208 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.462149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:58.872454 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.872425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:58.918514 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.918490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.918626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.918597    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:58.918680 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.918624    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.918680 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.918678    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:59.918663638 +0000 UTC m=+252.073032091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : configmap "kube-rbac-proxy" not found
Sep 15 18:54:58.918768 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.918740    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.918803 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:58.918788    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:54:59.918774972 +0000 UTC m=+252.073143418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:58.973479 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:58.973452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:54:59.926243 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:59.926208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:59.926422 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:54:59.926265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:54:59.926422 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:59.926345    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:59.926422 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:59.926382    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:54:59.926422 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:59.926419    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.926391847 +0000 UTC m=+254.080760292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:54:59.926576 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:54:59.926455    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:01.926441603 +0000 UTC m=+254.080810049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:00.127784 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.127741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:55:00.128123 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:00.127880    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 18:55:00.128123 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:00.127939    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs podName:7aff6fb0-49d5-4729-b743-fdf79154f869 nodeName:}" failed. No retries permitted until 2026-09-15 18:57:02.127923972 +0000 UTC m=+374.282292417 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs") pod "network-metrics-daemon-x77j7" (UID: "7aff6fb0-49d5-4729-b743-fdf79154f869") : secret "metrics-daemon-secret" not found
Sep 15 18:55:00.329519 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.329481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:00.329718 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.329546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:00.329718 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:00.329650    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 18:55:00.329718 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:00.329683    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:08.329669029 +0000 UTC m=+260.484037475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : configmap references non-existent config key: service-ca.crt
Sep 15 18:55:00.329718 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:00.329697    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:08.329691356 +0000 UTC m=+260.484059802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : secret "router-metrics-certs-default" not found
Sep 15 18:55:00.463032 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.463003    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tvshw"]
Sep 15 18:55:00.466121 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.466105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.468572 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.468551    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 18:55:00.468792 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.468776    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 18:55:00.468869 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.468832    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 18:55:00.469468 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.469449    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m2zvr\""
Sep 15 18:55:00.469589 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.469449    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 18:55:00.474601 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.474581    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tvshw"]
Sep 15 18:55:00.545685 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.545658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:00.601518 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.601452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:00.631673 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.631638    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cf1a30c-5999-4faa-9e8e-e7673748908d-signing-cabundle\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.631807 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.631686    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529xw\" (UniqueName: \"kubernetes.io/projected/7cf1a30c-5999-4faa-9e8e-e7673748908d-kube-api-access-529xw\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.631807 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.631733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cf1a30c-5999-4faa-9e8e-e7673748908d-signing-key\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.732138 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.732097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cf1a30c-5999-4faa-9e8e-e7673748908d-signing-cabundle\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.732272 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.732149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529xw\" (UniqueName: \"kubernetes.io/projected/7cf1a30c-5999-4faa-9e8e-e7673748908d-kube-api-access-529xw\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.732272 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.732177    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cf1a30c-5999-4faa-9e8e-e7673748908d-signing-key\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.732823 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.732800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cf1a30c-5999-4faa-9e8e-e7673748908d-signing-cabundle\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.734579 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.734557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cf1a30c-5999-4faa-9e8e-e7673748908d-signing-key\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.740119 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.740095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529xw\" (UniqueName: \"kubernetes.io/projected/7cf1a30c-5999-4faa-9e8e-e7673748908d-kube-api-access-529xw\") pod \"service-ca-568869fdc5-tvshw\" (UID: \"7cf1a30c-5999-4faa-9e8e-e7673748908d\") " pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:00.775850 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:00.775831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-tvshw"
Sep 15 18:55:01.039127 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.039091    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-tvshw"]
Sep 15 18:55:01.041913 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:01.041890    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf1a30c_5999_4faa_9e8e_e7673748908d.slice/crio-bfc1e5a425ec6f20e75d2781b2b38efe5ab6c3c660b34f1ad3e0801f9aeff60d WatchSource:0}: Error finding container bfc1e5a425ec6f20e75d2781b2b38efe5ab6c3c660b34f1ad3e0801f9aeff60d: Status 404 returned error can't find the container with id bfc1e5a425ec6f20e75d2781b2b38efe5ab6c3c660b34f1ad3e0801f9aeff60d
Sep 15 18:55:01.095895 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.095872    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-px5bd_5fbf53cb-b32a-4e88-b5c1-2959df28a398/dns-node-resolver/0.log"
Sep 15 18:55:01.457875 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.457796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:01.895891 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.895865    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-khm77_9c258100-5235-4aee-b9a7-7e97267b0cb0/node-ca/0.log"
Sep 15 18:55:01.942190 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.942159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:01.942324 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.942247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:01.942324 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:01.942273    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:55:01.942324 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:01.942323    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:05.942308226 +0000 UTC m=+258.096676677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:01.942428 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:01.942343    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:01.942428 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:01.942388    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:05.942376631 +0000 UTC m=+258.096745076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:01.975280 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:01.975251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-tvshw" event={"ID":"7cf1a30c-5999-4faa-9e8e-e7673748908d","Type":"ContainerStarted","Data":"bfc1e5a425ec6f20e75d2781b2b38efe5ab6c3c660b34f1ad3e0801f9aeff60d"}
Sep 15 18:55:02.498850 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:02.498822    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7jnzj_8a47024c-545b-4dc3-b464-77bc724c4d1d/serve-healthcheck-canary/0.log"
Sep 15 18:55:03.497052 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:03.497026    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-x9rlg_de912687-0dfa-4dd1-8562-1e01213aee3f/kube-storage-version-migrator-operator/0.log"
Sep 15 18:55:04.079263 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:04.079240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:04.986904 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:04.986858    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-tvshw" event={"ID":"7cf1a30c-5999-4faa-9e8e-e7673748908d","Type":"ContainerStarted","Data":"0949afbf6404bc9fb50463258bd67f6c25358fcce0bc07bdfc548261cdba452a"}
Sep 15 18:55:05.005009 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:05.004958    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-tvshw" podStartSLOduration=2.011061966 podStartE2EDuration="5.004935372s" totalImagesPullingTime="2.993873406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:05.003155421 +0000 UTC m=+257.157523890" watchObservedRunningTime="2026-09-15 18:55:05.004935372 +0000 UTC m=+257.159303842"
Sep 15 18:55:05.061870 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:05.061832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:05.971594 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:05.971561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:05.971983 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:05.971626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:05.971983 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:05.971726    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:05.971983 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:05.971755    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 18:55:05.971983 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:05.971788    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:13.971771792 +0000 UTC m=+266.126140255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:05.971983 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:05.971804    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:13.971796977 +0000 UTC m=+266.126165424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : configmap "kube-rbac-proxy" not found
Sep 15 18:55:06.100541 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:06.100508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:08.067827 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:08.067792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:08.168369 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:08.168335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:08.393876 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:08.393795    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:08.393876 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:08.393859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:08.394104 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:08.393957    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 18:55:08.394104 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:08.394008    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:24.39398461 +0000 UTC m=+276.548353062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : configmap references non-existent config key: service-ca.crt
Sep 15 18:55:08.394104 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:08.394032    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs podName:042f4fd4-4470-4b60-927d-37c53e0f25d7 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:24.394021581 +0000 UTC m=+276.548390033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs") pod "router-default-647fd6f876-2xxs8" (UID: "042f4fd4-4470-4b60-927d-37c53e0f25d7") : secret "router-metrics-certs-default" not found
Sep 15 18:55:08.562874 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:08.562845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:08.758948 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:08.758916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:10.112737 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:10.112703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:10.213050 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:10.213009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:10.559182 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:10.559150    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:10.759934 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:10.759902    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:11.360753 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:11.360727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:11.962592 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:11.962544    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:12.165910 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:12.165881    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:12.452356 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:12.452312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:12.554358 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:12.554322    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:12.761038 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:12.761005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:12.961916 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:12.961887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:13.965373 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:13.965338    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:14.039854 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:14.039822    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:14.039995 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:14.039883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:14.039995 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:14.039960    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:14.040106 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:14.040037    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls podName:5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:30.04001772 +0000 UTC m=+282.194386167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v6g27" (UID: "5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6") : secret "insights-runtime-extractor-tls" not found
Sep 15 18:55:14.040443 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:14.040423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:14.161985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:14.161954    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:14.561009 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:14.560975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:14.766292 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:14.766260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:15.358738 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:15.358703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:15.561718 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:15.561688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:15.758831 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:15.758803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:16.821273 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:16.821243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:16.924590 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:16.924561    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:17.023158 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:17.023120    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:17.324671 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:17.324643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:17.511829 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:17.511800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:17.560645 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:17.560607    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:17.900774 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:17.900747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:17.950422 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:17.950397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:18.710575 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:18.710548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:18.916704 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:18.916674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:19.326578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:19.326543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:19.519801 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:19.519773    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:20.324343 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:20.324293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:20.511386 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:20.511347    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:22.467839 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.467811    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-665c487469-v2k97"]
Sep 15 18:55:22.468183 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:22.467965    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-665c487469-v2k97" podUID="18c248e9-c09c-48da-b1bf-9ec0401841f5"
Sep 15 18:55:22.576211 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.576184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:22.579538 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.579518    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8cdd6fd85-rh95h"]
Sep 15 18:55:22.582024 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.582006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.592453 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.592430    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8cdd6fd85-rh95h"]
Sep 15 18:55:22.608067 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c2a232d-30d1-4941-af91-f2317ea2d488-ca-trust-extracted\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608182 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c2a232d-30d1-4941-af91-f2317ea2d488-registry-certificates\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608182 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-bound-sa-token\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608182 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4s67\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-kube-api-access-w4s67\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608182    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-registry-tls\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a232d-30d1-4941-af91-f2317ea2d488-trusted-ca\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c2a232d-30d1-4941-af91-f2317ea2d488-installation-pull-secrets\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.608327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.608300    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c2a232d-30d1-4941-af91-f2317ea2d488-image-registry-private-configuration\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.679109 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.679082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:22.709626 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c2a232d-30d1-4941-af91-f2317ea2d488-ca-trust-extracted\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709721 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c2a232d-30d1-4941-af91-f2317ea2d488-registry-certificates\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709721 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-bound-sa-token\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709721 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4s67\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-kube-api-access-w4s67\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709721 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-registry-tls\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709896 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a232d-30d1-4941-af91-f2317ea2d488-trusted-ca\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709896 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c2a232d-30d1-4941-af91-f2317ea2d488-installation-pull-secrets\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.709896 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.709809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c2a232d-30d1-4941-af91-f2317ea2d488-image-registry-private-configuration\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.710483 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.710460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c2a232d-30d1-4941-af91-f2317ea2d488-ca-trust-extracted\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.710629 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.710549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c2a232d-30d1-4941-af91-f2317ea2d488-registry-certificates\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.710931 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.710908    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c2a232d-30d1-4941-af91-f2317ea2d488-trusted-ca\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.712213 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.712174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c2a232d-30d1-4941-af91-f2317ea2d488-installation-pull-secrets\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.712296 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.712273    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c2a232d-30d1-4941-af91-f2317ea2d488-image-registry-private-configuration\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.712353 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.712332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-registry-tls\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.716942 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.716924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4s67\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-kube-api-access-w4s67\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.717051 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.716999    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c2a232d-30d1-4941-af91-f2317ea2d488-bound-sa-token\") pod \"image-registry-8cdd6fd85-rh95h\" (UID: \"8c2a232d-30d1-4941-af91-f2317ea2d488\") " pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:22.892489 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.892469    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwrs6\""
Sep 15 18:55:22.901310 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:22.901292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:23.036446 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.036419    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:55:23.042020 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.041999    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:55:23.112688 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112664    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-bound-sa-token\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.112786 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112696    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18c248e9-c09c-48da-b1bf-9ec0401841f5-ca-trust-extracted\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.112786 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112714    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gv4g\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-kube-api-access-9gv4g\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.112786 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112734    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-installation-pull-secrets\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.112786 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112757    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-trusted-ca\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.112786 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112779    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-certificates\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.113009 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112831    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-image-registry-private-configuration\") pod \"18c248e9-c09c-48da-b1bf-9ec0401841f5\" (UID: \"18c248e9-c09c-48da-b1bf-9ec0401841f5\") "
Sep 15 18:55:23.113009 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.112976    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c248e9-c09c-48da-b1bf-9ec0401841f5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:55:23.113165 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.113138    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18c248e9-c09c-48da-b1bf-9ec0401841f5-ca-trust-extracted\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:23.113296 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.113275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:55:23.113370 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.113344    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:55:23.114949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.114924    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:55:23.115020 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.114983    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:55:23.115218 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.115196    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-kube-api-access-9gv4g" (OuterVolumeSpecName: "kube-api-access-9gv4g") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "kube-api-access-9gv4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:55:23.115218 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.115199    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "18c248e9-c09c-48da-b1bf-9ec0401841f5" (UID: "18c248e9-c09c-48da-b1bf-9ec0401841f5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:55:23.168759 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.168711    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8cdd6fd85-rh95h"]
Sep 15 18:55:23.171732 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:23.171711    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2a232d_30d1_4941_af91_f2317ea2d488.slice/crio-1fe5422c1281700155be09a198475f237bef4ef865b7a7327cd1cd2542d9129c WatchSource:0}: Error finding container 1fe5422c1281700155be09a198475f237bef4ef865b7a7327cd1cd2542d9129c: Status 404 returned error can't find the container with id 1fe5422c1281700155be09a198475f237bef4ef865b7a7327cd1cd2542d9129c
Sep 15 18:55:23.213890 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.213870    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-installation-pull-secrets\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:23.213985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.213892    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-trusted-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:23.213985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.213906    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-certificates\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:23.213985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.213919    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18c248e9-c09c-48da-b1bf-9ec0401841f5-image-registry-private-configuration\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:23.213985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.213932    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-bound-sa-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:23.213985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:23.213944    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gv4g\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-kube-api-access-9gv4g\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:24.040847 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.040807    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h" event={"ID":"8c2a232d-30d1-4941-af91-f2317ea2d488","Type":"ContainerStarted","Data":"04d8662f9d1e858c1d6a14969003b8a315435f20615695fb9461b6a40e602587"}
Sep 15 18:55:24.040847 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.040833    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665c487469-v2k97"
Sep 15 18:55:24.041229 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.040898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h" event={"ID":"8c2a232d-30d1-4941-af91-f2317ea2d488","Type":"ContainerStarted","Data":"1fe5422c1281700155be09a198475f237bef4ef865b7a7327cd1cd2542d9129c"}
Sep 15 18:55:24.041229 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.041010    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:24.059492 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.059448    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h" podStartSLOduration=2.059435283 podStartE2EDuration="2.059435283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:24.05785658 +0000 UTC m=+276.212225049" watchObservedRunningTime="2026-09-15 18:55:24.059435283 +0000 UTC m=+276.213803751"
Sep 15 18:55:24.086146 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.086121    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-665c487469-v2k97"]
Sep 15 18:55:24.090724 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.090703    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-665c487469-v2k97"]
Sep 15 18:55:24.121606 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.121570    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18c248e9-c09c-48da-b1bf-9ec0401841f5-registry-tls\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:55:24.230245 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.230224    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c248e9-c09c-48da-b1bf-9ec0401841f5" path="/var/lib/kubelet/pods/18c248e9-c09c-48da-b1bf-9ec0401841f5/volumes"
Sep 15 18:55:24.423037 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.422977    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:24.423037 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.423035    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:24.423553 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.423532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/042f4fd4-4470-4b60-927d-37c53e0f25d7-service-ca-bundle\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:24.425061 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.425039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/042f4fd4-4470-4b60-927d-37c53e0f25d7-metrics-certs\") pod \"router-default-647fd6f876-2xxs8\" (UID: \"042f4fd4-4470-4b60-927d-37c53e0f25d7\") " pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:24.676403 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.676326    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:24.946787 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:24.946719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-647fd6f876-2xxs8"]
Sep 15 18:55:24.949568 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:24.949543    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042f4fd4_4470_4b60_927d_37c53e0f25d7.slice/crio-10c2a7e255b16433bd4bd3c5721adc4aa00f9f6164cd624757b468ad60d2c749 WatchSource:0}: Error finding container 10c2a7e255b16433bd4bd3c5721adc4aa00f9f6164cd624757b468ad60d2c749: Status 404 returned error can't find the container with id 10c2a7e255b16433bd4bd3c5721adc4aa00f9f6164cd624757b468ad60d2c749
Sep 15 18:55:25.045216 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:25.045178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-647fd6f876-2xxs8" event={"ID":"042f4fd4-4470-4b60-927d-37c53e0f25d7","Type":"ContainerStarted","Data":"477036aa52d803fff374934e2665a5c8ae3785ca9432550bbd3e51b22e27f90a"}
Sep 15 18:55:25.045556 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:25.045232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-647fd6f876-2xxs8" event={"ID":"042f4fd4-4470-4b60-927d-37c53e0f25d7","Type":"ContainerStarted","Data":"10c2a7e255b16433bd4bd3c5721adc4aa00f9f6164cd624757b468ad60d2c749"}
Sep 15 18:55:25.065407 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:25.065347    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-647fd6f876-2xxs8" podStartSLOduration=33.065333452 podStartE2EDuration="33.065333452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:25.063902797 +0000 UTC m=+277.218271267" watchObservedRunningTime="2026-09-15 18:55:25.065333452 +0000 UTC m=+277.219701920"
Sep 15 18:55:25.677059 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:25.677024    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:25.679511 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:25.679487    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:26.048430 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:26.048405    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:26.049539 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:26.049525    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-647fd6f876-2xxs8"
Sep 15 18:55:27.238926 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:27.238893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:27.338831 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:27.338801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:27.415194 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:27.415169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:27.937072 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:27.937024    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:28.663301 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:28.663271    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:28.719719 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:28.719689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:28.738001 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:28.737978    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-58p5d" podUID="c098a940-98ba-4e64-a691-e2cd0d6a41bc"
Sep 15 18:55:29.055780 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:29.055755    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-58p5d"
Sep 15 18:55:29.527130 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:29.527091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:29.592557 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:29.592537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:30.060813 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:30.060784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:30.063146 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:30.063124    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6g27\" (UID: \"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6\") " pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:30.295926 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:30.295891    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v6g27"
Sep 15 18:55:30.563745 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:30.563712    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v6g27"]
Sep 15 18:55:30.568297 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:30.568273    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f37fd67_89d8_42d6_9ce9_cded8dc1f7f6.slice/crio-1b2adce29fe521c45f4ee6e4f301246886b576ae7b495de3887f283f551f3d7a WatchSource:0}: Error finding container 1b2adce29fe521c45f4ee6e4f301246886b576ae7b495de3887f283f551f3d7a: Status 404 returned error can't find the container with id 1b2adce29fe521c45f4ee6e4f301246886b576ae7b495de3887f283f551f3d7a
Sep 15 18:55:31.063990 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.063957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6g27" event={"ID":"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6","Type":"ContainerStarted","Data":"093839f8329e2e9c5467b6be607f5982401d7eb30c5edb00d49173fdb509b89b"}
Sep 15 18:55:31.064337 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.064027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6g27" event={"ID":"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6","Type":"ContainerStarted","Data":"1b2adce29fe521c45f4ee6e4f301246886b576ae7b495de3887f283f551f3d7a"}
Sep 15 18:55:31.427687 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.427597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:31.458359 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.458333    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"]
Sep 15 18:55:31.460349 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.460334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.462552 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.462524    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 18:55:31.463505 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.463484    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 18:55:31.463593 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.463518    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 18:55:31.463593 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.463519    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 18:55:31.463593 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.463577    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 18:55:31.463884 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.463867    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mfqct\""
Sep 15 18:55:31.468530 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.468509    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"]
Sep 15 18:55:31.528733 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.528708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:31.573054 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.573026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2655aa5-cd43-4859-936f-8cc57b374ffe-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.573164 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.573075    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2655aa5-cd43-4859-936f-8cc57b374ffe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.573164 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.573106    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflqf\" (UniqueName: \"kubernetes.io/projected/c2655aa5-cd43-4859-936f-8cc57b374ffe-kube-api-access-kflqf\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.573164 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.573128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2655aa5-cd43-4859-936f-8cc57b374ffe-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.630061 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.630032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:31.674103 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.674074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2655aa5-cd43-4859-936f-8cc57b374ffe-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.674229 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.674122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2655aa5-cd43-4859-936f-8cc57b374ffe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.674229 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.674165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflqf\" (UniqueName: \"kubernetes.io/projected/c2655aa5-cd43-4859-936f-8cc57b374ffe-kube-api-access-kflqf\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.674229 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.674197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2655aa5-cd43-4859-936f-8cc57b374ffe-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.674896 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.674875    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2655aa5-cd43-4859-936f-8cc57b374ffe-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.676491 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.676473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2655aa5-cd43-4859-936f-8cc57b374ffe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.676571 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.676546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2655aa5-cd43-4859-936f-8cc57b374ffe-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.681297 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.681244    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflqf\" (UniqueName: \"kubernetes.io/projected/c2655aa5-cd43-4859-936f-8cc57b374ffe-kube-api-access-kflqf\") pod \"prometheus-operator-68bcd7f987-ccdtf\" (UID: \"c2655aa5-cd43-4859-936f-8cc57b374ffe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.770044 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.770015    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"
Sep 15 18:55:31.775165 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.775142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:55:31.777967 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:31.777937    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c098a940-98ba-4e64-a691-e2cd0d6a41bc-metrics-tls\") pod \"dns-default-58p5d\" (UID: \"c098a940-98ba-4e64-a691-e2cd0d6a41bc\") " pod="openshift-dns/dns-default-58p5d"
Sep 15 18:55:32.041494 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:32.041332    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf"]
Sep 15 18:55:32.044071 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:32.044044    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2655aa5_cd43_4859_936f_8cc57b374ffe.slice/crio-f6e2b31729335afe5cfc2df0079ee51883014fd7160c6b88af2ee822d7554a9d WatchSource:0}: Error finding container f6e2b31729335afe5cfc2df0079ee51883014fd7160c6b88af2ee822d7554a9d: Status 404 returned error can't find the container with id f6e2b31729335afe5cfc2df0079ee51883014fd7160c6b88af2ee822d7554a9d
Sep 15 18:55:32.061065 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:32.061034    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hj774\""
Sep 15 18:55:32.067196 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:32.067179    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-58p5d"
Sep 15 18:55:32.069035 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:32.069010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6g27" event={"ID":"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6","Type":"ContainerStarted","Data":"ccb473275a465f3a9488f2307c24502050c8706196053d71a9544b75d4668ffe"}
Sep 15 18:55:32.070221 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:32.070191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf" event={"ID":"c2655aa5-cd43-4859-936f-8cc57b374ffe","Type":"ContainerStarted","Data":"f6e2b31729335afe5cfc2df0079ee51883014fd7160c6b88af2ee822d7554a9d"}
Sep 15 18:55:32.336452 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:32.336384    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-58p5d"]
Sep 15 18:55:32.664032 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:32.663958    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc098a940_98ba_4e64_a691_e2cd0d6a41bc.slice/crio-003b3ad873487f00ecda079914f19c83a521affcfd23ddb811c0cf0d0fd8f154 WatchSource:0}: Error finding container 003b3ad873487f00ecda079914f19c83a521affcfd23ddb811c0cf0d0fd8f154: Status 404 returned error can't find the container with id 003b3ad873487f00ecda079914f19c83a521affcfd23ddb811c0cf0d0fd8f154
Sep 15 18:55:33.076635 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:33.076270    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6g27" event={"ID":"5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6","Type":"ContainerStarted","Data":"f6b838b090f277d257e98ddf07e45399e88c8d4efcca21f98342c1872b6bb848"}
Sep 15 18:55:33.078209 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:33.078169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-58p5d" event={"ID":"c098a940-98ba-4e64-a691-e2cd0d6a41bc","Type":"ContainerStarted","Data":"003b3ad873487f00ecda079914f19c83a521affcfd23ddb811c0cf0d0fd8f154"}
Sep 15 18:55:33.094518 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:33.094452    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v6g27" podStartSLOduration=33.076963245 podStartE2EDuration="35.094434485s" totalImagesPullingTime="2.01747124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 18:54:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:33.092648611 +0000 UTC m=+285.247017077" watchObservedRunningTime="2026-09-15 18:55:33.094434485 +0000 UTC m=+285.248802955"
Sep 15 18:55:33.939139 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:33.939057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:34.085424 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.085387    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf" event={"ID":"c2655aa5-cd43-4859-936f-8cc57b374ffe","Type":"ContainerStarted","Data":"a0bd688faa2bb3ce8d8f3f5f1940c06e507b84b56b2cb83b4dd74a622fa2c2d5"}
Sep 15 18:55:34.085859 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.085454    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf" event={"ID":"c2655aa5-cd43-4859-936f-8cc57b374ffe","Type":"ContainerStarted","Data":"b50acf8d92a50a3d7a52ae7da428c6f72875a0f4bcd2dd994fc5f1500a57fbdf"}
Sep 15 18:55:34.110453 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.110405    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-ccdtf" podStartSLOduration=1.498939116 podStartE2EDuration="3.110390921s" totalImagesPullingTime="1.611451805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:34.108591621 +0000 UTC m=+286.262960092" watchObservedRunningTime="2026-09-15 18:55:34.110390921 +0000 UTC m=+286.264759389"
Sep 15 18:55:34.526588 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.526516    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-794769b469-hvjlh"]
Sep 15 18:55:34.528350 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.528334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.530646 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.530623    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 18:55:34.530749 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.530675    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 18:55:34.530749 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.530694    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kbhdv\""
Sep 15 18:55:34.530992 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.530976    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 18:55:34.531054 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.530976    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 18:55:34.531228 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.531215    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 18:55:34.531386 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.531372    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 18:55:34.531572 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.531557    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 18:55:34.536077 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.536057    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 18:55:34.541361 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.541340    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794769b469-hvjlh"]
Sep 15 18:55:34.598119 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-serving-cert\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.598213 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-trusted-ca-bundle\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.598213 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29qb\" (UniqueName: \"kubernetes.io/projected/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-kube-api-access-r29qb\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.598213 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-oauth-config\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.598314 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-oauth-serving-cert\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.598314 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598281    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-service-ca\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.598378 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.598323    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-config\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.698947 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.698920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-service-ca\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699072 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.698954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-config\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699072 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.698996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-serving-cert\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699072 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699018    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-trusted-ca-bundle\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29qb\" (UniqueName: \"kubernetes.io/projected/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-kube-api-access-r29qb\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-oauth-config\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-oauth-serving-cert\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699653 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-service-ca\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.699951 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-config\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.700023 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.699985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-oauth-serving-cert\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.700080 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.700048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-trusted-ca-bundle\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.701484 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.701459    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-oauth-config\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.701571 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.701463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-serving-cert\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.708130 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.708100    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29qb\" (UniqueName: \"kubernetes.io/projected/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-kube-api-access-r29qb\") pod \"console-794769b469-hvjlh\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") " pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:34.837208 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:34.837153    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:35.094582 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.094492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-58p5d" event={"ID":"c098a940-98ba-4e64-a691-e2cd0d6a41bc","Type":"ContainerStarted","Data":"f6b11e7a7d8552a77889b3375fcf26cd013b00f4b1f073594ba880299359e3f3"}
Sep 15 18:55:35.095019 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.094581    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-58p5d" event={"ID":"c098a940-98ba-4e64-a691-e2cd0d6a41bc","Type":"ContainerStarted","Data":"7e56886d4bdb7527672fd63cdd3050f5d560b22d15c94f6d6fd9627b0dc16513"}
Sep 15 18:55:35.105715 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.105689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794769b469-hvjlh"]
Sep 15 18:55:35.108892 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:35.108840    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06f19fc_65f1_4af4_bb2c_bc06acf756a0.slice/crio-a3d036dec37cd3b5dff843b871b18360ededc9b65405b8b9b081ae89ec2eacc4 WatchSource:0}: Error finding container a3d036dec37cd3b5dff843b871b18360ededc9b65405b8b9b081ae89ec2eacc4: Status 404 returned error can't find the container with id a3d036dec37cd3b5dff843b871b18360ededc9b65405b8b9b081ae89ec2eacc4
Sep 15 18:55:35.109797 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.109689    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-58p5d" podStartSLOduration=252.519518832 podStartE2EDuration="4m14.109674545s" totalImagesPullingTime="1.590155713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:35.109075685 +0000 UTC m=+287.263444153" watchObservedRunningTime="2026-09-15 18:55:35.109674545 +0000 UTC m=+287.264043067"
Sep 15 18:55:35.544940 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.544909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:35.797200 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.797124    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"]
Sep 15 18:55:35.799568 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.799542    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:35.800015 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.799978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:35.805287 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.805065    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 18:55:35.805287 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.805089    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lddvc\""
Sep 15 18:55:35.805287 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.805125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 18:55:35.810749 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.810564    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rf469"]
Sep 15 18:55:35.812928 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.812900    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.813487 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.813425    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"]
Sep 15 18:55:35.818105 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.817525    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 18:55:35.818105 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.817671    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 18:55:35.818105 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.817704    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 18:55:35.818319 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.818223    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xf85x\""
Sep 15 18:55:35.898770 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.898740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:35.910031 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.909996    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vwbn\" (UniqueName: \"kubernetes.io/projected/503da95e-73a0-49a8-85e2-f1b38d92d85c-kube-api-access-7vwbn\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:35.910169 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmtg\" (UniqueName: \"kubernetes.io/projected/a893bf4b-87b6-46e1-8e2e-01e94f891f11-kube-api-access-ccmtg\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910169 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503da95e-73a0-49a8-85e2-f1b38d92d85c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:35.910282 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910171    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910282 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910197    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-accelerators-collector-config\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910282 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910223    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a893bf4b-87b6-46e1-8e2e-01e94f891f11-metrics-client-ca\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910421 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910278    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-wtmp\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910421 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:35.910421 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-tls\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910421 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910384    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-root\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910421 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-textfile\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910673 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-sys\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:35.910673 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:35.910465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.011038 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-wtmp\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-tls\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-root\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-textfile\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-sys\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.011227 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-wtmp\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011275    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-root\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:36.011315    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:36.011373    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-tls podName:a893bf4b-87b6-46e1-8e2e-01e94f891f11 nodeName:}" failed. No retries permitted until 2026-09-15 18:55:36.51135247 +0000 UTC m=+288.665720920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-tls") pod "node-exporter-rf469" (UID: "a893bf4b-87b6-46e1-8e2e-01e94f891f11") : secret "node-exporter-tls" not found
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:36.011384    2584 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:36.011432    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-tls podName:503da95e-73a0-49a8-85e2-f1b38d92d85c nodeName:}" failed. No retries permitted until 2026-09-15 18:55:36.511416557 +0000 UTC m=+288.665785008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-96b78" (UID: "503da95e-73a0-49a8-85e2-f1b38d92d85c") : secret "openshift-state-metrics-tls" not found
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vwbn\" (UniqueName: \"kubernetes.io/projected/503da95e-73a0-49a8-85e2-f1b38d92d85c-kube-api-access-7vwbn\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmtg\" (UniqueName: \"kubernetes.io/projected/a893bf4b-87b6-46e1-8e2e-01e94f891f11-kube-api-access-ccmtg\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a893bf4b-87b6-46e1-8e2e-01e94f891f11-sys\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503da95e-73a0-49a8-85e2-f1b38d92d85c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-textfile\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-accelerators-collector-config\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.011768 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.011633    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a893bf4b-87b6-46e1-8e2e-01e94f891f11-metrics-client-ca\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.012454 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.012238    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a893bf4b-87b6-46e1-8e2e-01e94f891f11-metrics-client-ca\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.012454 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.012281    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-accelerators-collector-config\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.012454 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.012307    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503da95e-73a0-49a8-85e2-f1b38d92d85c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.014327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.014303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.015529 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.015502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.019730 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.019661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vwbn\" (UniqueName: \"kubernetes.io/projected/503da95e-73a0-49a8-85e2-f1b38d92d85c-kube-api-access-7vwbn\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.020341 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.020305    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmtg\" (UniqueName: \"kubernetes.io/projected/a893bf4b-87b6-46e1-8e2e-01e94f891f11-kube-api-access-ccmtg\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.100418 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.100220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794769b469-hvjlh" event={"ID":"c06f19fc-65f1-4af4-bb2c-bc06acf756a0","Type":"ContainerStarted","Data":"a3d036dec37cd3b5dff843b871b18360ededc9b65405b8b9b081ae89ec2eacc4"}
Sep 15 18:55:36.100814 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.100414    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-58p5d"
Sep 15 18:55:36.118104 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.118074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:36.218478 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.218446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:36.400140 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.400062    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:36.502773 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.502740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:36.517726 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.517696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.517873 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.517789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-tls\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.520348 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.520322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a893bf4b-87b6-46e1-8e2e-01e94f891f11-node-exporter-tls\") pod \"node-exporter-rf469\" (UID: \"a893bf4b-87b6-46e1-8e2e-01e94f891f11\") " pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.520465 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.520426    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/503da95e-73a0-49a8-85e2-f1b38d92d85c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-96b78\" (UID: \"503da95e-73a0-49a8-85e2-f1b38d92d85c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.603079 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.603043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:36.714720 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.714643    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"
Sep 15 18:55:36.725688 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.725662    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rf469"
Sep 15 18:55:36.831466 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.831414    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:36.906570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:36.906535    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:37.004276 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:37.004209    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78"]
Sep 15 18:55:37.007193 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:55:37.007162    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503da95e_73a0_49a8_85e2_f1b38d92d85c.slice/crio-3c7e13dc1bd397efb1c61842252930760cd959bef7e37f0f252a92045f29736f WatchSource:0}: Error finding container 3c7e13dc1bd397efb1c61842252930760cd959bef7e37f0f252a92045f29736f: Status 404 returned error can't find the container with id 3c7e13dc1bd397efb1c61842252930760cd959bef7e37f0f252a92045f29736f
Sep 15 18:55:37.107558 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:37.107478    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf469" event={"ID":"a893bf4b-87b6-46e1-8e2e-01e94f891f11","Type":"ContainerStarted","Data":"a8668eced1fee0ed3c088db3b19bc66156056a842f959d4674a6a1a0134d6024"}
Sep 15 18:55:37.109357 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:37.109326    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78" event={"ID":"503da95e-73a0-49a8-85e2-f1b38d92d85c","Type":"ContainerStarted","Data":"22b45a3fcfd70c76ca907f716a8fa5ef1b5a3c47dd9476f502759eeaab369238"}
Sep 15 18:55:37.109485 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:37.109398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78" event={"ID":"503da95e-73a0-49a8-85e2-f1b38d92d85c","Type":"ContainerStarted","Data":"3c7e13dc1bd397efb1c61842252930760cd959bef7e37f0f252a92045f29736f"}
Sep 15 18:55:37.371683 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:37.371571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:38.704731 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:38.704697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:38.772513 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:38.772484    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:39.120503 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.120211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794769b469-hvjlh" event={"ID":"c06f19fc-65f1-4af4-bb2c-bc06acf756a0","Type":"ContainerStarted","Data":"0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694"}
Sep 15 18:55:39.122594 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.122567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78" event={"ID":"503da95e-73a0-49a8-85e2-f1b38d92d85c","Type":"ContainerStarted","Data":"fa0207799c22cb0a1299b535e093b1f9570d723c4ff32ba9f19501a2f26287a7"}
Sep 15 18:55:39.122728 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.122650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78" event={"ID":"503da95e-73a0-49a8-85e2-f1b38d92d85c","Type":"ContainerStarted","Data":"872d07e4c2065a415521db654f18857c21c4aef62f4cbfd6fda0458d81dbf61e"}
Sep 15 18:55:39.136447 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.136399    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-794769b469-hvjlh" podStartSLOduration=2.109170957 podStartE2EDuration="5.136380521s" totalImagesPullingTime="3.027209564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:39.135385773 +0000 UTC m=+291.289754242" watchObservedRunningTime="2026-09-15 18:55:39.136380521 +0000 UTC m=+291.290748990"
Sep 15 18:55:39.154124 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.154078    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-96b78" podStartSLOduration=3.329388664 podStartE2EDuration="4.154059393s" totalImagesPullingTime="824.670729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:39.153245527 +0000 UTC m=+291.307613995" watchObservedRunningTime="2026-09-15 18:55:39.154059393 +0000 UTC m=+291.308427864"
Sep 15 18:55:39.299711 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.299640    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:39.607571 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:39.607543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:40.129015 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:40.128979    2584 generic.go:356] "Generic (PLEG): container finished" podID="a893bf4b-87b6-46e1-8e2e-01e94f891f11" containerID="02dbfb7a4966c07acb30677766476c25afb43be599c869c0148d6cd6cd7bbecb" exitCode=0
Sep 15 18:55:40.129377 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:40.129057    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf469" event={"ID":"a893bf4b-87b6-46e1-8e2e-01e94f891f11","Type":"ContainerDied","Data":"02dbfb7a4966c07acb30677766476c25afb43be599c869c0148d6cd6cd7bbecb"}
Sep 15 18:55:40.580707 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:40.580676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:40.674703 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:40.674670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:41.135401 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:41.135365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf469" event={"ID":"a893bf4b-87b6-46e1-8e2e-01e94f891f11","Type":"ContainerStarted","Data":"31bb78a6ff9594e662fe8dab208b29f935b691b4a28275423b1689dc2dd01e0e"}
Sep 15 18:55:41.135890 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:41.135429    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rf469" event={"ID":"a893bf4b-87b6-46e1-8e2e-01e94f891f11","Type":"ContainerStarted","Data":"65b8bd2566e9b07d7ee2244849958876447fb9bd50e7683a928494bae13cdcdd"}
Sep 15 18:55:41.153024 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:41.152974    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rf469" podStartSLOduration=3.310759098 podStartE2EDuration="6.152956184s" totalImagesPullingTime="2.780088035s" totalInitContainerRuntime="62.109051ms" isStatefulPod=true podCreationTimestamp="2026-09-15 18:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:55:41.151282824 +0000 UTC m=+293.305651292" watchObservedRunningTime="2026-09-15 18:55:41.152956184 +0000 UTC m=+293.307324654"
Sep 15 18:55:41.227580 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:41.227543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:41.944934 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:41.944899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:42.228647 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:55:42.228538    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-x77j7" podUID="7aff6fb0-49d5-4729-b743-fdf79154f869"
Sep 15 18:55:42.906223 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:42.906185    2584 patch_prober.go:28] interesting pod/image-registry-8cdd6fd85-rh95h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 18:55:42.906393 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:42.906241    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h" podUID="8c2a232d-30d1-4941-af91-f2317ea2d488" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 18:55:43.448476 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:43.448428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:43.548300 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:43.548264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:43.596683 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:43.596648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:44.837798 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:44.837764    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:44.838183 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:44.837842    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:44.842584 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:44.842561    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:45.049762 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:45.049730    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8cdd6fd85-rh95h"
Sep 15 18:55:45.152903 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:45.152824    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:55:45.181302 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:45.181273    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:45.272818 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:45.272789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:45.374332 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:45.374301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:45.609640 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:45.609593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:46.111866 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:46.111835    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-58p5d"
Sep 15 18:55:46.309666 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:46.309633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:46.359798 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:46.359769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:47.125118 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:47.125087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:48.190969 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:48.190944    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 18:55:51.716777 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:51.716742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:51.774197 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:51.774167    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:53.826242 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:53.826200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:53.926649 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:53.926599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:54.227462 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:54.227367    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:55:56.123805 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:56.123772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:55:56.183665 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:55:56.183635    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:00.755472 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:00.755439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:00.857378 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:00.857348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:01.848063 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:01.848028    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794769b469-hvjlh"]
Sep 15 18:56:01.954813 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:01.954774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:02.011838 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:02.011792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:02.300871 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:02.300838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:02.906683 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:02.906584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:02.979150 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:02.979121    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:05.343458 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:05.343424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:05.397292 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:05.397255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:17.268459 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:17.268422    2584 generic.go:356] "Generic (PLEG): container finished" podID="de912687-0dfa-4dd1-8562-1e01213aee3f" containerID="5b059a79cebfe9b8b457ad5c1bda5720e40afd51b8e1e32c08149a54b8c1e821" exitCode=0
Sep 15 18:56:17.268891 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:17.268496    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg" event={"ID":"de912687-0dfa-4dd1-8562-1e01213aee3f","Type":"ContainerDied","Data":"5b059a79cebfe9b8b457ad5c1bda5720e40afd51b8e1e32c08149a54b8c1e821"}
Sep 15 18:56:17.268891 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:17.268821    2584 scope.go:117] "RemoveContainer" containerID="5b059a79cebfe9b8b457ad5c1bda5720e40afd51b8e1e32c08149a54b8c1e821"
Sep 15 18:56:18.276743 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:18.276714    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-x9rlg" event={"ID":"de912687-0dfa-4dd1-8562-1e01213aee3f","Type":"ContainerStarted","Data":"502701ac2ecce3a06f2bbbb76daa8fd79bfd320452421257c5b602763146f6a3"}
Sep 15 18:56:20.328121 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:20.328088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:20.382108 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:20.382075    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:26.873329 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:26.873295    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-794769b469-hvjlh" podUID="c06f19fc-65f1-4af4-bb2c-bc06acf756a0" containerName="console" containerID="cri-o://0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694" gracePeriod=15
Sep 15 18:56:27.108319 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.108296    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:56:27.195745 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195668    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r29qb\" (UniqueName: \"kubernetes.io/projected/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-kube-api-access-r29qb\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.195745 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195705    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-serving-cert\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.195745 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195722    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-trusted-ca-bundle\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.196000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195755    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-service-ca\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.196000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195851    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-oauth-serving-cert\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.196000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195898    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-oauth-config\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.196000 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.195947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-config\") pod \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\" (UID: \"c06f19fc-65f1-4af4-bb2c-bc06acf756a0\") "
Sep 15 18:56:27.196178 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.196144    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:27.196178 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.196154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:27.196280 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.196221    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:27.196505 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.196482    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-config" (OuterVolumeSpecName: "console-config") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 18:56:27.197936 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.197912    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-kube-api-access-r29qb" (OuterVolumeSpecName: "kube-api-access-r29qb") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "kube-api-access-r29qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:56:27.198085 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.198065    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:56:27.198144 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.198084    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c06f19fc-65f1-4af4-bb2c-bc06acf756a0" (UID: "c06f19fc-65f1-4af4-bb2c-bc06acf756a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 18:56:27.296918 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296889    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.296918 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296913    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29qb\" (UniqueName: \"kubernetes.io/projected/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-kube-api-access-r29qb\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.296918 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296923    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-serving-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.297113 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296931    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-trusted-ca-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.297113 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296939    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.297113 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296947    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-oauth-serving-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.297113 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.296955    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c06f19fc-65f1-4af4-bb2c-bc06acf756a0-console-oauth-config\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:56:27.309133 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.309108    2584 generic.go:356] "Generic (PLEG): container finished" podID="c06f19fc-65f1-4af4-bb2c-bc06acf756a0" containerID="0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694" exitCode=0
Sep 15 18:56:27.309254 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.309146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794769b469-hvjlh" event={"ID":"c06f19fc-65f1-4af4-bb2c-bc06acf756a0","Type":"ContainerDied","Data":"0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694"}
Sep 15 18:56:27.309254 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.309157    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794769b469-hvjlh"
Sep 15 18:56:27.309254 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.309188    2584 scope.go:117] "RemoveContainer" containerID="0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694"
Sep 15 18:56:27.309404 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.309178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794769b469-hvjlh" event={"ID":"c06f19fc-65f1-4af4-bb2c-bc06acf756a0","Type":"ContainerDied","Data":"a3d036dec37cd3b5dff843b871b18360ededc9b65405b8b9b081ae89ec2eacc4"}
Sep 15 18:56:27.317247 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.317226    2584 scope.go:117] "RemoveContainer" containerID="0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694"
Sep 15 18:56:27.317523 ip-10-0-143-252 kubenswrapper[2584]: E0915 18:56:27.317500    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694\": container with ID starting with 0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694 not found: ID does not exist" containerID="0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694"
Sep 15 18:56:27.317600 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.317529    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694"} err="failed to get container status \"0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694\": rpc error: code = NotFound desc = could not find container \"0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694\": container with ID starting with 0dc4a1eec3bc76d31fb0d4ba81d433313a24b6b57ea4c65f9d9ae13ab50e2694 not found: ID does not exist"
Sep 15 18:56:27.329185 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.329162    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794769b469-hvjlh"]
Sep 15 18:56:27.332372 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:27.332352    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-794769b469-hvjlh"]
Sep 15 18:56:28.231656 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:28.231598    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c06f19fc-65f1-4af4-bb2c-bc06acf756a0" path="/var/lib/kubelet/pods/c06f19fc-65f1-4af4-bb2c-bc06acf756a0/volumes"
Sep 15 18:56:28.416550 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:28.416520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:28.468715 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:28.468689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:33.765178 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:33.765143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:35.921144 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:35.921112    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:55.017779 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:55.017735    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:55.462980 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:55.462847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:58.150929 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:58.150885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:56:58.246202 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:56:58.246170    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:00.072796 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.072754    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cbd8d687-qs44c"]
Sep 15 18:57:00.073245 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.073131    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c06f19fc-65f1-4af4-bb2c-bc06acf756a0" containerName="console"
Sep 15 18:57:00.073245 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.073148    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c06f19fc-65f1-4af4-bb2c-bc06acf756a0" containerName="console"
Sep 15 18:57:00.073245 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.073221    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c06f19fc-65f1-4af4-bb2c-bc06acf756a0" containerName="console"
Sep 15 18:57:00.078112 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.078089    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.080372 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.080349    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 18:57:00.080506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.080398    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-j6g8s\""
Sep 15 18:57:00.080506 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.080418    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 18:57:00.080681 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.080509    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 18:57:00.080681 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.080653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 18:57:00.080823 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.080695    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 18:57:00.086731 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.086713    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 18:57:00.092672 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.092653    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cbd8d687-qs44c"]
Sep 15 18:57:00.149288 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-serving-certs-ca-bundle\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149481 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149481 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-secret-telemeter-client\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149481 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-telemeter-client-tls\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149481 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149406    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s7qd\" (UniqueName: \"kubernetes.io/projected/45e2eef4-16ff-4a18-a686-479ab2c7d981-kube-api-access-7s7qd\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149481 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149481 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-metrics-client-ca\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.149736 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.149518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-federate-client-tls\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.249985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.249953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.249985 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.249983    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-metrics-client-ca\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250008    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-federate-client-tls\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-serving-certs-ca-bundle\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-secret-telemeter-client\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250284 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-telemeter-client-tls\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250540 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s7qd\" (UniqueName: \"kubernetes.io/projected/45e2eef4-16ff-4a18-a686-479ab2c7d981-kube-api-access-7s7qd\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250836 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250813    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-metrics-client-ca\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.250942 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.250818    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-serving-certs-ca-bundle\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.251040 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.251019    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45e2eef4-16ff-4a18-a686-479ab2c7d981-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.252696 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.252676    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-federate-client-tls\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.252838 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.252817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.252879 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.252850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-secret-telemeter-client\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.252913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.252880    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45e2eef4-16ff-4a18-a686-479ab2c7d981-telemeter-client-tls\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.257809 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.257791    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s7qd\" (UniqueName: \"kubernetes.io/projected/45e2eef4-16ff-4a18-a686-479ab2c7d981-kube-api-access-7s7qd\") pod \"telemeter-client-cbd8d687-qs44c\" (UID: \"45e2eef4-16ff-4a18-a686-479ab2c7d981\") " pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.388771 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.388684    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c"
Sep 15 18:57:00.660958 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.660888    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cbd8d687-qs44c"]
Sep 15 18:57:00.664084 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:57:00.664059    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e2eef4_16ff_4a18_a686_479ab2c7d981.slice/crio-a6a8f0c67e922d8d856d3808bdaf1bb5ed341f9a647eae6c5dbfadd3a0aaac58 WatchSource:0}: Error finding container a6a8f0c67e922d8d856d3808bdaf1bb5ed341f9a647eae6c5dbfadd3a0aaac58: Status 404 returned error can't find the container with id a6a8f0c67e922d8d856d3808bdaf1bb5ed341f9a647eae6c5dbfadd3a0aaac58
Sep 15 18:57:00.666465 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:00.666447    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 18:57:01.433560 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:01.433516    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c" event={"ID":"45e2eef4-16ff-4a18-a686-479ab2c7d981","Type":"ContainerStarted","Data":"a6a8f0c67e922d8d856d3808bdaf1bb5ed341f9a647eae6c5dbfadd3a0aaac58"}
Sep 15 18:57:02.164685 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:02.164646    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:57:02.167564 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:02.167536    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7aff6fb0-49d5-4729-b743-fdf79154f869-metrics-certs\") pod \"network-metrics-daemon-x77j7\" (UID: \"7aff6fb0-49d5-4729-b743-fdf79154f869\") " pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:57:02.331351 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:02.331320    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bc8c\""
Sep 15 18:57:02.339096 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:02.339068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x77j7"
Sep 15 18:57:02.925856 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:02.925663    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x77j7"]
Sep 15 18:57:02.928212 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:57:02.928178    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aff6fb0_49d5_4729_b743_fdf79154f869.slice/crio-f9cf36adaff3750ef8c590230789c0595575db37bea2eb9db0c99764f77771a4 WatchSource:0}: Error finding container f9cf36adaff3750ef8c590230789c0595575db37bea2eb9db0c99764f77771a4: Status 404 returned error can't find the container with id f9cf36adaff3750ef8c590230789c0595575db37bea2eb9db0c99764f77771a4
Sep 15 18:57:03.443702 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:03.443603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c" event={"ID":"45e2eef4-16ff-4a18-a686-479ab2c7d981","Type":"ContainerStarted","Data":"7c90044df616649b6f871ad00eb3dd018b718cd1ec4a44447a55215b646a98f2"}
Sep 15 18:57:03.445022 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:03.444991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x77j7" event={"ID":"7aff6fb0-49d5-4729-b743-fdf79154f869","Type":"ContainerStarted","Data":"f9cf36adaff3750ef8c590230789c0595575db37bea2eb9db0c99764f77771a4"}
Sep 15 18:57:04.451256 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.451174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c" event={"ID":"45e2eef4-16ff-4a18-a686-479ab2c7d981","Type":"ContainerStarted","Data":"61eb3145c7c4177dc3f04dbd235388f1e083ef3a8d47adc1fef76f6e535fa2fe"}
Sep 15 18:57:04.451256 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.451237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c" event={"ID":"45e2eef4-16ff-4a18-a686-479ab2c7d981","Type":"ContainerStarted","Data":"b66444fee6e0e31e0766996d639e133bb8857c5fb04cfca7dc3ed144aa1ccc5c"}
Sep 15 18:57:04.453281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.453255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x77j7" event={"ID":"7aff6fb0-49d5-4729-b743-fdf79154f869","Type":"ContainerStarted","Data":"bfe0d43a06e95d3f117c247022c78b8a1c2f6260b41d8c914c60f9be5d91e53e"}
Sep 15 18:57:04.453388 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.453304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x77j7" event={"ID":"7aff6fb0-49d5-4729-b743-fdf79154f869","Type":"ContainerStarted","Data":"67ce43d398d264b792d2f43916a61fe4368dbecde39eb050eaa7ce6fad6d2b03"}
Sep 15 18:57:04.507303 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.507255    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x77j7" podStartSLOduration=375.35880214 podStartE2EDuration="6m16.507240611s" totalImagesPullingTime="1.148438471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:04.50571861 +0000 UTC m=+376.660087079" watchObservedRunningTime="2026-09-15 18:57:04.507240611 +0000 UTC m=+376.661609078"
Sep 15 18:57:04.507720 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.507689    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cbd8d687-qs44c" podStartSLOduration=1.152971301 podStartE2EDuration="4.507680412s" totalImagesPullingTime="3.354709111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:57:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:57:04.48807657 +0000 UTC m=+376.642445038" watchObservedRunningTime="2026-09-15 18:57:04.507680412 +0000 UTC m=+376.662048880"
Sep 15 18:57:04.694688 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.694654    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:04.773277 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.773244    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:04.890056 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:04.890018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:05.474904 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:05.474868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:05.577958 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:05.577930    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:06.831584 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:06.831545    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:06.888577 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:06.888540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:10.214587 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:10.214557    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:10.267502 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:10.267471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:15.596897 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:15.596861    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:15.918406 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:15.918316    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:22.224434 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:22.224390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:22.283417 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:22.283388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:26.646074 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:26.646042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:26.811197 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:26.811167    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:27.008088 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:27.008056    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:35.334045 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:35.334011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:35.434279 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:35.434250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:35.488240 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:35.488212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:57:56.242240 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:57:56.242197    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:03.956846 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:03.956814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:31.462799 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:31.462757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:31.557724 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:31.557697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:32.091357 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.091320    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"]
Sep 15 18:58:32.094437 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.094422    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.096711 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.096688    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 18:58:32.097480 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.097457    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nljx\""
Sep 15 18:58:32.097594 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.097490    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 18:58:32.102926 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.102906    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"]
Sep 15 18:58:32.177206 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.177164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.177344 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.177215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.177344 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.177239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhvd\" (UniqueName: \"kubernetes.io/projected/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-kube-api-access-cvhvd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.278280 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.278254    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.278400 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.278291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.278471 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.278447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhvd\" (UniqueName: \"kubernetes.io/projected/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-kube-api-access-cvhvd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.278683 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.278665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.278787 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.278722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.286164 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.286147    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhvd\" (UniqueName: \"kubernetes.io/projected/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-kube-api-access-cvhvd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.403483 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.403413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:32.672495 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.672322    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"]
Sep 15 18:58:32.675117 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:58:32.675091    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b39a8b5_6e34_45d7_92c5_dbda49a707f2.slice/crio-f6c4c4d420301d8195496859c62d8f2ea9389de9b79af8631d09886162f2fcbc WatchSource:0}: Error finding container f6c4c4d420301d8195496859c62d8f2ea9389de9b79af8631d09886162f2fcbc: Status 404 returned error can't find the container with id f6c4c4d420301d8195496859c62d8f2ea9389de9b79af8631d09886162f2fcbc
Sep 15 18:58:32.779390 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:32.779357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7" event={"ID":"9b39a8b5-6e34-45d7-92c5-dbda49a707f2","Type":"ContainerStarted","Data":"f6c4c4d420301d8195496859c62d8f2ea9389de9b79af8631d09886162f2fcbc"}
Sep 15 18:58:38.804776 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:38.804740    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7" event={"ID":"9b39a8b5-6e34-45d7-92c5-dbda49a707f2","Type":"ContainerStarted","Data":"09404d19ae91945da5162f0f48c3777583dfe86c64e78bbefbaa9f4614ff91f8"}
Sep 15 18:58:39.809874 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:39.809834    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerID="09404d19ae91945da5162f0f48c3777583dfe86c64e78bbefbaa9f4614ff91f8" exitCode=0
Sep 15 18:58:39.810246 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:39.809884    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7" event={"ID":"9b39a8b5-6e34-45d7-92c5-dbda49a707f2","Type":"ContainerDied","Data":"09404d19ae91945da5162f0f48c3777583dfe86c64e78bbefbaa9f4614ff91f8"}
Sep 15 18:58:42.824926 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:42.824887    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerID="79ac96548d994fe8f4187a5d5ec88b26f8074837957b9327467753b98eda9ebf" exitCode=0
Sep 15 18:58:42.825288 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:42.824942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7" event={"ID":"9b39a8b5-6e34-45d7-92c5-dbda49a707f2","Type":"ContainerDied","Data":"79ac96548d994fe8f4187a5d5ec88b26f8074837957b9327467753b98eda9ebf"}
Sep 15 18:58:49.854280 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:49.854244    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerID="6b138f177c9bc074e8aa4c62cdf0ac572a33b779b734383368bf03b6aeebe9e2" exitCode=0
Sep 15 18:58:49.854659 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:49.854330    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7" event={"ID":"9b39a8b5-6e34-45d7-92c5-dbda49a707f2","Type":"ContainerDied","Data":"6b138f177c9bc074e8aa4c62cdf0ac572a33b779b734383368bf03b6aeebe9e2"}
Sep 15 18:58:50.977851 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:50.977819    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:51.138842 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.138760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-bundle\") pod \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") "
Sep 15 18:58:51.138842 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.138811    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-util\") pod \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") "
Sep 15 18:58:51.139022 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.138855    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvhvd\" (UniqueName: \"kubernetes.io/projected/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-kube-api-access-cvhvd\") pod \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\" (UID: \"9b39a8b5-6e34-45d7-92c5-dbda49a707f2\") "
Sep 15 18:58:51.139422 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.139395    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-bundle" (OuterVolumeSpecName: "bundle") pod "9b39a8b5-6e34-45d7-92c5-dbda49a707f2" (UID: "9b39a8b5-6e34-45d7-92c5-dbda49a707f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:58:51.141047 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.141017    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-kube-api-access-cvhvd" (OuterVolumeSpecName: "kube-api-access-cvhvd") pod "9b39a8b5-6e34-45d7-92c5-dbda49a707f2" (UID: "9b39a8b5-6e34-45d7-92c5-dbda49a707f2"). InnerVolumeSpecName "kube-api-access-cvhvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:58:51.142889 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.142859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-util" (OuterVolumeSpecName: "util") pod "9b39a8b5-6e34-45d7-92c5-dbda49a707f2" (UID: "9b39a8b5-6e34-45d7-92c5-dbda49a707f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:58:51.239776 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.239736    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:58:51.239776 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.239769    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:58:51.239776 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.239781    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhvd\" (UniqueName: \"kubernetes.io/projected/9b39a8b5-6e34-45d7-92c5-dbda49a707f2-kube-api-access-cvhvd\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:58:51.863578 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.863552    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7"
Sep 15 18:58:51.863761 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.863557    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kt7r7" event={"ID":"9b39a8b5-6e34-45d7-92c5-dbda49a707f2","Type":"ContainerDied","Data":"f6c4c4d420301d8195496859c62d8f2ea9389de9b79af8631d09886162f2fcbc"}
Sep 15 18:58:51.863761 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:51.863694    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c4c4d420301d8195496859c62d8f2ea9389de9b79af8631d09886162f2fcbc"
Sep 15 18:58:53.203281 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:53.203246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:54.074693 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:54.074658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:55.225994 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.225961    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"]
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226219    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="extract"
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226233    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="extract"
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226243    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="util"
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226248    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="util"
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226263    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="pull"
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226268    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="pull"
Sep 15 18:58:55.226331 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.226318    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b39a8b5-6e34-45d7-92c5-dbda49a707f2" containerName="extract"
Sep 15 18:58:55.268412 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.268383    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"]
Sep 15 18:58:55.268557 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.268493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.273058 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.273040    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r965d\""
Sep 15 18:58:55.273058 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.273051    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 18:58:55.273321 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.273303    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:58:55.368973 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.368946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ce6297-bfed-4b3c-8fce-e7432380c7c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-q8szl\" (UID: \"99ce6297-bfed-4b3c-8fce-e7432380c7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.369096 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.368994    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tsc\" (UniqueName: \"kubernetes.io/projected/99ce6297-bfed-4b3c-8fce-e7432380c7c8-kube-api-access-l5tsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-q8szl\" (UID: \"99ce6297-bfed-4b3c-8fce-e7432380c7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.469450 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.469424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ce6297-bfed-4b3c-8fce-e7432380c7c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-q8szl\" (UID: \"99ce6297-bfed-4b3c-8fce-e7432380c7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.469584 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.469469    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tsc\" (UniqueName: \"kubernetes.io/projected/99ce6297-bfed-4b3c-8fce-e7432380c7c8-kube-api-access-l5tsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-q8szl\" (UID: \"99ce6297-bfed-4b3c-8fce-e7432380c7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.469838 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.469820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ce6297-bfed-4b3c-8fce-e7432380c7c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-q8szl\" (UID: \"99ce6297-bfed-4b3c-8fce-e7432380c7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.478886 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.478830    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tsc\" (UniqueName: \"kubernetes.io/projected/99ce6297-bfed-4b3c-8fce-e7432380c7c8-kube-api-access-l5tsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-q8szl\" (UID: \"99ce6297-bfed-4b3c-8fce-e7432380c7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.577283 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.577261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"
Sep 15 18:58:55.866375 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.866352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl"]
Sep 15 18:58:55.869097 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:58:55.869069    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ce6297_bfed_4b3c_8fce_e7432380c7c8.slice/crio-a806a5824c0258100db50397eb1ad2e9d792778286555added4d003c6e8da3be WatchSource:0}: Error finding container a806a5824c0258100db50397eb1ad2e9d792778286555added4d003c6e8da3be: Status 404 returned error can't find the container with id a806a5824c0258100db50397eb1ad2e9d792778286555added4d003c6e8da3be
Sep 15 18:58:55.878380 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:55.878358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl" event={"ID":"99ce6297-bfed-4b3c-8fce-e7432380c7c8","Type":"ContainerStarted","Data":"a806a5824c0258100db50397eb1ad2e9d792778286555added4d003c6e8da3be"}
Sep 15 18:58:56.811740 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:56.811690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:56.911659 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:56.911623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:58.607783 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:58.607749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:58.707465 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:58.707432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:58.895370 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:58.895284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl" event={"ID":"99ce6297-bfed-4b3c-8fce-e7432380c7c8","Type":"ContainerStarted","Data":"6f08bec420ce32d279c3b4b07e1d8cbac40ba953e5f60c3f2404cda966fd585b"}
Sep 15 18:58:58.934570 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:58.934510    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q8szl" podStartSLOduration=1.940092422 podStartE2EDuration="3.934494932s" totalImagesPullingTime="1.99440251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:58:58.933746219 +0000 UTC m=+491.088114687" watchObservedRunningTime="2026-09-15 18:58:58.934494932 +0000 UTC m=+491.088863401"
Sep 15 18:58:59.269442 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:59.269407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:58:59.373853 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:58:59.373824    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:00.630907 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.630877    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"]
Sep 15 18:59:00.634099 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.634079    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.636454 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.636431    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 18:59:00.636556 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.636432    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 18:59:00.637278 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.637260    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nljx\""
Sep 15 18:59:00.642693 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.642672    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"]
Sep 15 18:59:00.652377 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.652356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:00.810903 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.810872    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.810903 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.810907    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgpz\" (UniqueName: \"kubernetes.io/projected/4f729eec-04ce-4a8c-b12a-74402da1fe68-kube-api-access-lfgpz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.811051 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.810924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.911930 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.911856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.911930 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.911894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgpz\" (UniqueName: \"kubernetes.io/projected/4f729eec-04ce-4a8c-b12a-74402da1fe68-kube-api-access-lfgpz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.911930 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.911913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.912291 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.912269    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.912355 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.912296    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.919643 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.919622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgpz\" (UniqueName: \"kubernetes.io/projected/4f729eec-04ce-4a8c-b12a-74402da1fe68-kube-api-access-lfgpz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:00.943528 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:00.943505    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:01.013891 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.013860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:01.191789 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.191715    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"]
Sep 15 18:59:01.193909 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.193885    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.196126 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.196099    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 18:59:01.196249 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.196156    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6b7m8\""
Sep 15 18:59:01.196305 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.196265    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 18:59:01.203939 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.203913    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"]
Sep 15 18:59:01.211064 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.211040    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"]
Sep 15 18:59:01.214894 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:59:01.214867    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f729eec_04ce_4a8c_b12a_74402da1fe68.slice/crio-3f91a39beb81c8f0a86a07d05af33b9c0ebafbd884aaba352e282bbbe4fc2ff3 WatchSource:0}: Error finding container 3f91a39beb81c8f0a86a07d05af33b9c0ebafbd884aaba352e282bbbe4fc2ff3: Status 404 returned error can't find the container with id 3f91a39beb81c8f0a86a07d05af33b9c0ebafbd884aaba352e282bbbe4fc2ff3
Sep 15 18:59:01.315764 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.315733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da5c1018-50ee-4011-8e53-b60635b70cb8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lpzqz\" (UID: \"da5c1018-50ee-4011-8e53-b60635b70cb8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.315894 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.315770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8x29\" (UniqueName: \"kubernetes.io/projected/da5c1018-50ee-4011-8e53-b60635b70cb8-kube-api-access-q8x29\") pod \"cert-manager-cainjector-559cdd8c87-lpzqz\" (UID: \"da5c1018-50ee-4011-8e53-b60635b70cb8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.416539 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.416508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da5c1018-50ee-4011-8e53-b60635b70cb8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lpzqz\" (UID: \"da5c1018-50ee-4011-8e53-b60635b70cb8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.416539 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.416540    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8x29\" (UniqueName: \"kubernetes.io/projected/da5c1018-50ee-4011-8e53-b60635b70cb8-kube-api-access-q8x29\") pod \"cert-manager-cainjector-559cdd8c87-lpzqz\" (UID: \"da5c1018-50ee-4011-8e53-b60635b70cb8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.424678 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.424651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da5c1018-50ee-4011-8e53-b60635b70cb8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lpzqz\" (UID: \"da5c1018-50ee-4011-8e53-b60635b70cb8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.424832 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.424814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8x29\" (UniqueName: \"kubernetes.io/projected/da5c1018-50ee-4011-8e53-b60635b70cb8-kube-api-access-q8x29\") pod \"cert-manager-cainjector-559cdd8c87-lpzqz\" (UID: \"da5c1018-50ee-4011-8e53-b60635b70cb8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.515512 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.515487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"
Sep 15 18:59:01.781048 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.780966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz"]
Sep 15 18:59:01.783771 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:59:01.783742    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda5c1018_50ee_4011_8e53_b60635b70cb8.slice/crio-5413d5d4b6b7961678aa93d35ea44860f7aee726fd2536dcf4169256f3b79ab6 WatchSource:0}: Error finding container 5413d5d4b6b7961678aa93d35ea44860f7aee726fd2536dcf4169256f3b79ab6: Status 404 returned error can't find the container with id 5413d5d4b6b7961678aa93d35ea44860f7aee726fd2536dcf4169256f3b79ab6
Sep 15 18:59:01.908823 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.908789    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz" event={"ID":"da5c1018-50ee-4011-8e53-b60635b70cb8","Type":"ContainerStarted","Data":"5413d5d4b6b7961678aa93d35ea44860f7aee726fd2536dcf4169256f3b79ab6"}
Sep 15 18:59:01.910234 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.910208    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerID="a4de5fff706bcec946de67a9e2bb97f4bb38a155c8d6283f3534852f6d4910d3" exitCode=0
Sep 15 18:59:01.910351 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.910249    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj" event={"ID":"4f729eec-04ce-4a8c-b12a-74402da1fe68","Type":"ContainerDied","Data":"a4de5fff706bcec946de67a9e2bb97f4bb38a155c8d6283f3534852f6d4910d3"}
Sep 15 18:59:01.910351 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:01.910278    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj" event={"ID":"4f729eec-04ce-4a8c-b12a-74402da1fe68","Type":"ContainerStarted","Data":"3f91a39beb81c8f0a86a07d05af33b9c0ebafbd884aaba352e282bbbe4fc2ff3"}
Sep 15 18:59:04.606752 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:04.606713    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:04.706931 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:04.706884    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:05.930987 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:05.930949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz" event={"ID":"da5c1018-50ee-4011-8e53-b60635b70cb8","Type":"ContainerStarted","Data":"1c3658b95b302ef411d33a27fd07f3ed03c98b834c691447aa1499d40d802a97"}
Sep 15 18:59:05.932713 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:05.932683    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerID="7cc0e6e73c01997bef4e4010ee3c49ab062cbe9626e0fe73ad2697c40bd3d5e7" exitCode=0
Sep 15 18:59:05.932841 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:05.932730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj" event={"ID":"4f729eec-04ce-4a8c-b12a-74402da1fe68","Type":"ContainerDied","Data":"7cc0e6e73c01997bef4e4010ee3c49ab062cbe9626e0fe73ad2697c40bd3d5e7"}
Sep 15 18:59:05.948106 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:05.948061    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lpzqz" podStartSLOduration=1.724642385 podStartE2EDuration="4.948045186s" totalImagesPullingTime="3.223402801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:05.947730947 +0000 UTC m=+498.102099417" watchObservedRunningTime="2026-09-15 18:59:05.948045186 +0000 UTC m=+498.102413655"
Sep 15 18:59:06.938923 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:06.938890    2584 generic.go:356] "Generic (PLEG): container finished" podID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerID="9388007863bd7e5c5de87bb0078c1e673f3458deee3051043262621ebec526ee" exitCode=0
Sep 15 18:59:06.939301 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:06.938975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj" event={"ID":"4f729eec-04ce-4a8c-b12a-74402da1fe68","Type":"ContainerDied","Data":"9388007863bd7e5c5de87bb0078c1e673f3458deee3051043262621ebec526ee"}
Sep 15 18:59:08.057551 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.057529    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:08.169540 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.169512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-bundle\") pod \"4f729eec-04ce-4a8c-b12a-74402da1fe68\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") "
Sep 15 18:59:08.169706 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.169550    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfgpz\" (UniqueName: \"kubernetes.io/projected/4f729eec-04ce-4a8c-b12a-74402da1fe68-kube-api-access-lfgpz\") pod \"4f729eec-04ce-4a8c-b12a-74402da1fe68\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") "
Sep 15 18:59:08.169706 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.169578    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-util\") pod \"4f729eec-04ce-4a8c-b12a-74402da1fe68\" (UID: \"4f729eec-04ce-4a8c-b12a-74402da1fe68\") "
Sep 15 18:59:08.169946 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.169927    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-bundle" (OuterVolumeSpecName: "bundle") pod "4f729eec-04ce-4a8c-b12a-74402da1fe68" (UID: "4f729eec-04ce-4a8c-b12a-74402da1fe68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:08.171534 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.171509    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f729eec-04ce-4a8c-b12a-74402da1fe68-kube-api-access-lfgpz" (OuterVolumeSpecName: "kube-api-access-lfgpz") pod "4f729eec-04ce-4a8c-b12a-74402da1fe68" (UID: "4f729eec-04ce-4a8c-b12a-74402da1fe68"). InnerVolumeSpecName "kube-api-access-lfgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:59:08.173878 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.173848    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-util" (OuterVolumeSpecName: "util") pod "4f729eec-04ce-4a8c-b12a-74402da1fe68" (UID: "4f729eec-04ce-4a8c-b12a-74402da1fe68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:08.270388 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.270365    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:08.270388 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.270388    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f729eec-04ce-4a8c-b12a-74402da1fe68-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:08.270509 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.270402    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfgpz\" (UniqueName: \"kubernetes.io/projected/4f729eec-04ce-4a8c-b12a-74402da1fe68-kube-api-access-lfgpz\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:08.949173 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.949133    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj" event={"ID":"4f729eec-04ce-4a8c-b12a-74402da1fe68","Type":"ContainerDied","Data":"3f91a39beb81c8f0a86a07d05af33b9c0ebafbd884aaba352e282bbbe4fc2ff3"}
Sep 15 18:59:08.949173 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.949178    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fr4sxj"
Sep 15 18:59:08.949381 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:08.949191    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f91a39beb81c8f0a86a07d05af33b9c0ebafbd884aaba352e282bbbe4fc2ff3"
Sep 15 18:59:09.375052 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:09.375016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:09.474363 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:09.474333    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:16.205976 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:16.205946    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:16.300796 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:16.300768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:18.224493 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224457    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"]
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224732    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="pull"
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224743    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="pull"
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224755    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="util"
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224760    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="util"
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224771    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="extract"
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224777    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="extract"
Sep 15 18:59:18.224882 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.224820    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f729eec-04ce-4a8c-b12a-74402da1fe68" containerName="extract"
Sep 15 18:59:18.229167 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.229143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.231378 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.231350    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rl4j4\""
Sep 15 18:59:18.231378 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.231350    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 18:59:18.232276 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.232257    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 18:59:18.239008 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.238990    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"]
Sep 15 18:59:18.241519 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.241501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0169b61f-90ff-45b2-a6fa-39021ee6ffff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gw8gw\" (UID: \"0169b61f-90ff-45b2-a6fa-39021ee6ffff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.241598 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.241574    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gg6\" (UniqueName: \"kubernetes.io/projected/0169b61f-90ff-45b2-a6fa-39021ee6ffff-kube-api-access-42gg6\") pod \"openshift-lws-operator-fd8ccff4c-gw8gw\" (UID: \"0169b61f-90ff-45b2-a6fa-39021ee6ffff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.346052 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.342296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gg6\" (UniqueName: \"kubernetes.io/projected/0169b61f-90ff-45b2-a6fa-39021ee6ffff-kube-api-access-42gg6\") pod \"openshift-lws-operator-fd8ccff4c-gw8gw\" (UID: \"0169b61f-90ff-45b2-a6fa-39021ee6ffff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.346052 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.342379    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0169b61f-90ff-45b2-a6fa-39021ee6ffff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gw8gw\" (UID: \"0169b61f-90ff-45b2-a6fa-39021ee6ffff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.346052 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.342807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0169b61f-90ff-45b2-a6fa-39021ee6ffff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gw8gw\" (UID: \"0169b61f-90ff-45b2-a6fa-39021ee6ffff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.352585 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.352558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gg6\" (UniqueName: \"kubernetes.io/projected/0169b61f-90ff-45b2-a6fa-39021ee6ffff-kube-api-access-42gg6\") pod \"openshift-lws-operator-fd8ccff4c-gw8gw\" (UID: \"0169b61f-90ff-45b2-a6fa-39021ee6ffff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.537724 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.537697    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"
Sep 15 18:59:18.804486 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.804462    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw"]
Sep 15 18:59:18.807048 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:59:18.807017    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0169b61f_90ff_45b2_a6fa_39021ee6ffff.slice/crio-f23a8f078accc8d6fbeda493d3f87539e8a617a6331d463c56b8899796c19c4d WatchSource:0}: Error finding container f23a8f078accc8d6fbeda493d3f87539e8a617a6331d463c56b8899796c19c4d: Status 404 returned error can't find the container with id f23a8f078accc8d6fbeda493d3f87539e8a617a6331d463c56b8899796c19c4d
Sep 15 18:59:18.989946 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:18.989911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw" event={"ID":"0169b61f-90ff-45b2-a6fa-39021ee6ffff","Type":"ContainerStarted","Data":"f23a8f078accc8d6fbeda493d3f87539e8a617a6331d463c56b8899796c19c4d"}
Sep 15 18:59:21.000712 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:21.000627    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw" event={"ID":"0169b61f-90ff-45b2-a6fa-39021ee6ffff","Type":"ContainerStarted","Data":"9795e84713d97bf1a66d25352e0dd52c76420cb94a0469393327f8c7637fa37a"}
Sep 15 18:59:21.016201 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:21.016143    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gw8gw" podStartSLOduration=1.141419979 podStartE2EDuration="3.01612375s" totalImagesPullingTime="1.874703771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:21.015005226 +0000 UTC m=+513.169373715" watchObservedRunningTime="2026-09-15 18:59:21.01612375 +0000 UTC m=+513.170492221"
Sep 15 18:59:21.744913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:21.744876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:21.842959 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:21.842923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:24.283758 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:24.283723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:25.821204 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.821167    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"]
Sep 15 18:59:25.823657 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.823633    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.825673 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.825647    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 18:59:25.826492 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.826475    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nljx\""
Sep 15 18:59:25.826577 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.826528    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 18:59:25.833195 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.833167    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"]
Sep 15 18:59:25.892590 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.892565    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.892694 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.892606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.892694 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.892647    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrld\" (UniqueName: \"kubernetes.io/projected/c0b3dad9-c4ed-44fd-87ca-631a2449c458-kube-api-access-4qrld\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.992952 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.992924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.993042 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.992965    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.993042 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.992991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrld\" (UniqueName: \"kubernetes.io/projected/c0b3dad9-c4ed-44fd-87ca-631a2449c458-kube-api-access-4qrld\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.993311 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.993293    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:25.993377 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:25.993322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:26.000320 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:26.000294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrld\" (UniqueName: \"kubernetes.io/projected/c0b3dad9-c4ed-44fd-87ca-631a2449c458-kube-api-access-4qrld\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:26.136540 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:26.136465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:26.403968 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:26.403945    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"]
Sep 15 18:59:26.406019 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:59:26.405987    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b3dad9_c4ed_44fd_87ca_631a2449c458.slice/crio-0d0dc687863b486816ab33f9ab0c3ca812dff7e57ae441cf3a11af7585f1fc4b WatchSource:0}: Error finding container 0d0dc687863b486816ab33f9ab0c3ca812dff7e57ae441cf3a11af7585f1fc4b: Status 404 returned error can't find the container with id 0d0dc687863b486816ab33f9ab0c3ca812dff7e57ae441cf3a11af7585f1fc4b
Sep 15 18:59:27.025209 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:27.025178    2584 generic.go:356] "Generic (PLEG): container finished" podID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerID="0f1688baa79f1aee4943a350607e42b0e628dad83863db70434cc848a5244434" exitCode=0
Sep 15 18:59:27.025514 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:27.025266    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5" event={"ID":"c0b3dad9-c4ed-44fd-87ca-631a2449c458","Type":"ContainerDied","Data":"0f1688baa79f1aee4943a350607e42b0e628dad83863db70434cc848a5244434"}
Sep 15 18:59:27.025514 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:27.025320    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5" event={"ID":"c0b3dad9-c4ed-44fd-87ca-631a2449c458","Type":"ContainerStarted","Data":"0d0dc687863b486816ab33f9ab0c3ca812dff7e57ae441cf3a11af7585f1fc4b"}
Sep 15 18:59:28.032698 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:28.032667    2584 generic.go:356] "Generic (PLEG): container finished" podID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerID="419bc2ad86eeadff8ab8c4d5abb4c3d7995d219473097db3db87f1e69e49c257" exitCode=0
Sep 15 18:59:28.033079 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:28.032754    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5" event={"ID":"c0b3dad9-c4ed-44fd-87ca-631a2449c458","Type":"ContainerDied","Data":"419bc2ad86eeadff8ab8c4d5abb4c3d7995d219473097db3db87f1e69e49c257"}
Sep 15 18:59:28.517949 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:28.517920    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:28.582884 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:28.582853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:29.039175 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:29.039144    2584 generic.go:356] "Generic (PLEG): container finished" podID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerID="c49043f6da926e109bd77a195690716fa3bd8653468cc0fc84f19f943918ee9c" exitCode=0
Sep 15 18:59:29.039580 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:29.039221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5" event={"ID":"c0b3dad9-c4ed-44fd-87ca-631a2449c458","Type":"ContainerDied","Data":"c49043f6da926e109bd77a195690716fa3bd8653468cc0fc84f19f943918ee9c"}
Sep 15 18:59:30.159796 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.159769    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:30.226131 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.226102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qrld\" (UniqueName: \"kubernetes.io/projected/c0b3dad9-c4ed-44fd-87ca-631a2449c458-kube-api-access-4qrld\") pod \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") "
Sep 15 18:59:30.226131 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.226136    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-bundle\") pod \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") "
Sep 15 18:59:30.226368 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.226156    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-util\") pod \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\" (UID: \"c0b3dad9-c4ed-44fd-87ca-631a2449c458\") "
Sep 15 18:59:30.226906 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.226817    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-bundle" (OuterVolumeSpecName: "bundle") pod "c0b3dad9-c4ed-44fd-87ca-631a2449c458" (UID: "c0b3dad9-c4ed-44fd-87ca-631a2449c458"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:30.228285 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.228249    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b3dad9-c4ed-44fd-87ca-631a2449c458-kube-api-access-4qrld" (OuterVolumeSpecName: "kube-api-access-4qrld") pod "c0b3dad9-c4ed-44fd-87ca-631a2449c458" (UID: "c0b3dad9-c4ed-44fd-87ca-631a2449c458"). InnerVolumeSpecName "kube-api-access-4qrld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:59:30.234551 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.234505    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-util" (OuterVolumeSpecName: "util") pod "c0b3dad9-c4ed-44fd-87ca-631a2449c458" (UID: "c0b3dad9-c4ed-44fd-87ca-631a2449c458"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:30.327444 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.327365    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrld\" (UniqueName: \"kubernetes.io/projected/c0b3dad9-c4ed-44fd-87ca-631a2449c458-kube-api-access-4qrld\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:30.327444 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.327398    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:30.327444 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:30.327412    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0b3dad9-c4ed-44fd-87ca-631a2449c458-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:31.049771 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:31.049738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5" event={"ID":"c0b3dad9-c4ed-44fd-87ca-631a2449c458","Type":"ContainerDied","Data":"0d0dc687863b486816ab33f9ab0c3ca812dff7e57ae441cf3a11af7585f1fc4b"}
Sep 15 18:59:31.049957 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:31.049775    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnzbh5"
Sep 15 18:59:31.049957 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:31.049785    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d0dc687863b486816ab33f9ab0c3ca812dff7e57ae441cf3a11af7585f1fc4b"
Sep 15 18:59:34.848913 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:34.848879    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:36.441947 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.441911    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"]
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442168    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="util"
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442178    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="util"
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442188    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="pull"
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442194    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="pull"
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442211    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="extract"
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442216    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="extract"
Sep 15 18:59:36.442362 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.442262    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b3dad9-c4ed-44fd-87ca-631a2449c458" containerName="extract"
Sep 15 18:59:36.444709 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.444689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.447265 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.447231    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 18:59:36.447391 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.447297    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nljx\""
Sep 15 18:59:36.448070 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.448053    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 18:59:36.454197 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.454177    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"]
Sep 15 18:59:36.478273 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.478248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl522\" (UniqueName: \"kubernetes.io/projected/11786c5e-12ce-4ff8-b61b-89b2de4daed0-kube-api-access-dl522\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.478381 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.478289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.478425 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.478388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.578996 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.578958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.579168 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.579007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl522\" (UniqueName: \"kubernetes.io/projected/11786c5e-12ce-4ff8-b61b-89b2de4daed0-kube-api-access-dl522\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.579168 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.579125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.579400 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.579379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.579464 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.579408    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.586389 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.586370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl522\" (UniqueName: \"kubernetes.io/projected/11786c5e-12ce-4ff8-b61b-89b2de4daed0-kube-api-access-dl522\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:36.753459 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:36.753433    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:37.042433 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:37.042402    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"]
Sep 15 18:59:37.043920 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:59:37.043895    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11786c5e_12ce_4ff8_b61b_89b2de4daed0.slice/crio-53ecf2fdbe388cb76c99a9cbdd5c4be2fc2fa36dbe6bb8e95e9bc84324c11b35 WatchSource:0}: Error finding container 53ecf2fdbe388cb76c99a9cbdd5c4be2fc2fa36dbe6bb8e95e9bc84324c11b35: Status 404 returned error can't find the container with id 53ecf2fdbe388cb76c99a9cbdd5c4be2fc2fa36dbe6bb8e95e9bc84324c11b35
Sep 15 18:59:37.077031 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:37.076998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8" event={"ID":"11786c5e-12ce-4ff8-b61b-89b2de4daed0","Type":"ContainerStarted","Data":"53ecf2fdbe388cb76c99a9cbdd5c4be2fc2fa36dbe6bb8e95e9bc84324c11b35"}
Sep 15 18:59:37.687637 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:37.687586    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:37.785157 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:37.785126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:38.083039 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.083003    2584 generic.go:356] "Generic (PLEG): container finished" podID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerID="3a37dedff3220675ce96f8eb34b813604b3a196f4856c9eebbd0301788bb6eca" exitCode=0
Sep 15 18:59:38.083199 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.083083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8" event={"ID":"11786c5e-12ce-4ff8-b61b-89b2de4daed0","Type":"ContainerDied","Data":"3a37dedff3220675ce96f8eb34b813604b3a196f4856c9eebbd0301788bb6eca"}
Sep 15 18:59:38.259148 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.259116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:38.540947 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.540919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:38.645279 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.645249    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:38.658106 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.658086    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"]
Sep 15 18:59:38.659942 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.659926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.662354 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.662335    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 18:59:38.662455 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.662369    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 18:59:38.662580 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.662555    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 18:59:38.662701 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.662559    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdkzs\""
Sep 15 18:59:38.662701 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.662667    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 18:59:38.676377 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.676357    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"]
Sep 15 18:59:38.694916 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.694894    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5czn\" (UniqueName: \"kubernetes.io/projected/6ab30393-65d5-4511-989c-826b66e83554-kube-api-access-z5czn\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.695247 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.694932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ab30393-65d5-4511-989c-826b66e83554-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.695247 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.694952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ab30393-65d5-4511-989c-826b66e83554-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.695247 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.694979    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ab30393-65d5-4511-989c-826b66e83554-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.695247 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.695029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ab30393-65d5-4511-989c-826b66e83554-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.791091 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.791019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:38.796146 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.796126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5czn\" (UniqueName: \"kubernetes.io/projected/6ab30393-65d5-4511-989c-826b66e83554-kube-api-access-z5czn\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.796203 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.796158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ab30393-65d5-4511-989c-826b66e83554-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.796203 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.796174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ab30393-65d5-4511-989c-826b66e83554-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.796310 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.796202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ab30393-65d5-4511-989c-826b66e83554-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.796406 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.796375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ab30393-65d5-4511-989c-826b66e83554-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.796579 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.796560    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ab30393-65d5-4511-989c-826b66e83554-manifests\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.798479 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.798455    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ab30393-65d5-4511-989c-826b66e83554-tmp\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.798564 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.798528    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ab30393-65d5-4511-989c-826b66e83554-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.798630 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.798564    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ab30393-65d5-4511-989c-826b66e83554-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.803400 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.803376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5czn\" (UniqueName: \"kubernetes.io/projected/6ab30393-65d5-4511-989c-826b66e83554-kube-api-access-z5czn\") pod \"opendatahub-operator-controller-manager-6d9df7495-t8z4v\" (UID: \"6ab30393-65d5-4511-989c-826b66e83554\") " pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:38.894462 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.894428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:38.969635 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:38.969592    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:39.011327 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:39.011297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:39.094385 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:39.094310    2584 generic.go:356] "Generic (PLEG): container finished" podID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerID="598d1eecc271973c36831ebda3f8a31c54149801d8800cd3fb7bf827782579a5" exitCode=0
Sep 15 18:59:39.094385 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:39.094346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8" event={"ID":"11786c5e-12ce-4ff8-b61b-89b2de4daed0","Type":"ContainerDied","Data":"598d1eecc271973c36831ebda3f8a31c54149801d8800cd3fb7bf827782579a5"}
Sep 15 18:59:39.113525 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:39.113496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:39.207199 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:39.207157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:39.242860 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:39.242830    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"]
Sep 15 18:59:39.286337 ip-10-0-143-252 kubenswrapper[2584]: W0915 18:59:39.286315    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab30393_65d5_4511_989c_826b66e83554.slice/crio-864137f6a20e90fc8e51cdb28f4be83425b61242541de8ba23c18c2aa8337555 WatchSource:0}: Error finding container 864137f6a20e90fc8e51cdb28f4be83425b61242541de8ba23c18c2aa8337555: Status 404 returned error can't find the container with id 864137f6a20e90fc8e51cdb28f4be83425b61242541de8ba23c18c2aa8337555
Sep 15 18:59:40.101514 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:40.101479    2584 generic.go:356] "Generic (PLEG): container finished" podID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerID="f1e87e9415353c0cb7c2886d0190c0c5d92bdef858fb8b95f3083dd7b1217f7a" exitCode=0
Sep 15 18:59:40.101946 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:40.101573    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8" event={"ID":"11786c5e-12ce-4ff8-b61b-89b2de4daed0","Type":"ContainerDied","Data":"f1e87e9415353c0cb7c2886d0190c0c5d92bdef858fb8b95f3083dd7b1217f7a"}
Sep 15 18:59:40.102871 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:40.102847    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v" event={"ID":"6ab30393-65d5-4511-989c-826b66e83554","Type":"ContainerStarted","Data":"864137f6a20e90fc8e51cdb28f4be83425b61242541de8ba23c18c2aa8337555"}
Sep 15 18:59:41.225086 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.225064    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:41.317999 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.317974    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-bundle\") pod \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") "
Sep 15 18:59:41.318144 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.318012    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl522\" (UniqueName: \"kubernetes.io/projected/11786c5e-12ce-4ff8-b61b-89b2de4daed0-kube-api-access-dl522\") pod \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") "
Sep 15 18:59:41.318144 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.318059    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-util\") pod \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\" (UID: \"11786c5e-12ce-4ff8-b61b-89b2de4daed0\") "
Sep 15 18:59:41.318811 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.318783    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-bundle" (OuterVolumeSpecName: "bundle") pod "11786c5e-12ce-4ff8-b61b-89b2de4daed0" (UID: "11786c5e-12ce-4ff8-b61b-89b2de4daed0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:41.320119 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.320094    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11786c5e-12ce-4ff8-b61b-89b2de4daed0-kube-api-access-dl522" (OuterVolumeSpecName: "kube-api-access-dl522") pod "11786c5e-12ce-4ff8-b61b-89b2de4daed0" (UID: "11786c5e-12ce-4ff8-b61b-89b2de4daed0"). InnerVolumeSpecName "kube-api-access-dl522". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 18:59:41.326600 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.326559    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-util" (OuterVolumeSpecName: "util") pod "11786c5e-12ce-4ff8-b61b-89b2de4daed0" (UID: "11786c5e-12ce-4ff8-b61b-89b2de4daed0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 18:59:41.419333 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.419281    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:41.419333 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.419303    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11786c5e-12ce-4ff8-b61b-89b2de4daed0-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:41.419333 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:41.419312    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl522\" (UniqueName: \"kubernetes.io/projected/11786c5e-12ce-4ff8-b61b-89b2de4daed0-kube-api-access-dl522\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 18:59:42.115921 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:42.115890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8" event={"ID":"11786c5e-12ce-4ff8-b61b-89b2de4daed0","Type":"ContainerDied","Data":"53ecf2fdbe388cb76c99a9cbdd5c4be2fc2fa36dbe6bb8e95e9bc84324c11b35"}
Sep 15 18:59:42.116079 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:42.115938    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53ecf2fdbe388cb76c99a9cbdd5c4be2fc2fa36dbe6bb8e95e9bc84324c11b35"
Sep 15 18:59:42.116079 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:42.115899    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0285l8"
Sep 15 18:59:45.134417 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:45.134380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v" event={"ID":"6ab30393-65d5-4511-989c-826b66e83554","Type":"ContainerStarted","Data":"46888ac5b053527d8a05a98beb243d5aa96089ad8598a5df9cd1fd705bd35bfc"}
Sep 15 18:59:46.140018 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:46.139983    2584 generic.go:356] "Generic (PLEG): container finished" podID="6ab30393-65d5-4511-989c-826b66e83554" containerID="46888ac5b053527d8a05a98beb243d5aa96089ad8598a5df9cd1fd705bd35bfc" exitCode=0
Sep 15 18:59:46.140459 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:46.140025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v" event={"ID":"6ab30393-65d5-4511-989c-826b66e83554","Type":"ContainerDied","Data":"46888ac5b053527d8a05a98beb243d5aa96089ad8598a5df9cd1fd705bd35bfc"}
Sep 15 18:59:47.146387 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:47.146301    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v" event={"ID":"6ab30393-65d5-4511-989c-826b66e83554","Type":"ContainerStarted","Data":"42ab6e81e068e7da0f48762ea4a216d19ce1002468fc13e66122643edb844858"}
Sep 15 18:59:47.146765 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:47.146543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:47.169202 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:47.169154    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v" podStartSLOduration=2.149472819 podStartE2EDuration="9.169139303s" totalImagesPullingTime="5.979693451s" totalInitContainerRuntime="1.039973033s" isStatefulPod=false podCreationTimestamp="2026-09-15 18:59:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 18:59:47.167004856 +0000 UTC m=+539.321373345" watchObservedRunningTime="2026-09-15 18:59:47.169139303 +0000 UTC m=+539.323507771"
Sep 15 18:59:58.153018 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:58.152985    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d9df7495-t8z4v"
Sep 15 18:59:59.705731 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:59.705694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 18:59:59.812209 ip-10-0-143-252 kubenswrapper[2584]: I0915 18:59:59.812178    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:05.225103 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:05.225069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:10.303946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.303915    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:10.433252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433218    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"]
Sep 15 19:00:10.433542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433530    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="extract"
Sep 15 19:00:10.433589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433543    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="extract"
Sep 15 19:00:10.433589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433557    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="util"
Sep 15 19:00:10.433589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433563    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="util"
Sep 15 19:00:10.433589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433576    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="pull"
Sep 15 19:00:10.433589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433581    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="pull"
Sep 15 19:00:10.433785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.433638    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="11786c5e-12ce-4ff8-b61b-89b2de4daed0" containerName="extract"
Sep 15 19:00:10.443864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.443835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.444627 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.444584    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"]
Sep 15 19:00:10.446182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.446155    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fmg74\""
Sep 15 19:00:10.446436 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.446420    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 19:00:10.446548 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.446531    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 19:00:10.551864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.551833    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1612dd-403c-4e3b-a679-23748bd36b1a-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.551864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.551865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1612dd-403c-4e3b-a679-23748bd36b1a-tmp\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.552116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.551959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stt6l\" (UniqueName: \"kubernetes.io/projected/2a1612dd-403c-4e3b-a679-23748bd36b1a-kube-api-access-stt6l\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.653087 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.652990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stt6l\" (UniqueName: \"kubernetes.io/projected/2a1612dd-403c-4e3b-a679-23748bd36b1a-kube-api-access-stt6l\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.653087 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.653052    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1612dd-403c-4e3b-a679-23748bd36b1a-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.653087 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.653075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1612dd-403c-4e3b-a679-23748bd36b1a-tmp\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.653372 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:10.653201    2584 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 15 19:00:10.653372 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:10.653285    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1612dd-403c-4e3b-a679-23748bd36b1a-tls-certs podName:2a1612dd-403c-4e3b-a679-23748bd36b1a nodeName:}" failed. No retries permitted until 2026-09-15 19:00:11.153266113 +0000 UTC m=+563.307634572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2a1612dd-403c-4e3b-a679-23748bd36b1a-tls-certs") pod "kube-auth-proxy-65488ff6c7-b8fc4" (UID: "2a1612dd-403c-4e3b-a679-23748bd36b1a") : secret "kube-auth-proxy-tls" not found
Sep 15 19:00:10.655310 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.655288    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1612dd-403c-4e3b-a679-23748bd36b1a-tmp\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.661414 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.661390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stt6l\" (UniqueName: \"kubernetes.io/projected/2a1612dd-403c-4e3b-a679-23748bd36b1a-kube-api-access-stt6l\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:10.815197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.815164    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sxr2k"]
Sep 15 19:00:10.818645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.818599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:10.820981 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.820962    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 15 19:00:10.821067 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.821052    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6n8qz\""
Sep 15 19:00:10.826361 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.826341    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sxr2k"]
Sep 15 19:00:10.955687 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.955569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b322b894-053a-4e46-8f24-2c72a523cf69-cert\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:10.955687 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:10.955650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5c9d\" (UniqueName: \"kubernetes.io/projected/b322b894-053a-4e46-8f24-2c72a523cf69-kube-api-access-j5c9d\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.056937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.056903    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b322b894-053a-4e46-8f24-2c72a523cf69-cert\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.057101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.056951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5c9d\" (UniqueName: \"kubernetes.io/projected/b322b894-053a-4e46-8f24-2c72a523cf69-kube-api-access-j5c9d\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.057101 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:11.057064    2584 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 15 19:00:11.057176 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:11.057129    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322b894-053a-4e46-8f24-2c72a523cf69-cert podName:b322b894-053a-4e46-8f24-2c72a523cf69 nodeName:}" failed. No retries permitted until 2026-09-15 19:00:11.557113202 +0000 UTC m=+563.711481652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b322b894-053a-4e46-8f24-2c72a523cf69-cert") pod "odh-model-controller-7998c67dc4-sxr2k" (UID: "b322b894-053a-4e46-8f24-2c72a523cf69") : secret "odh-model-controller-webhook-cert" not found
Sep 15 19:00:11.072307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.072282    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5c9d\" (UniqueName: \"kubernetes.io/projected/b322b894-053a-4e46-8f24-2c72a523cf69-kube-api-access-j5c9d\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.157659 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.157623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1612dd-403c-4e3b-a679-23748bd36b1a-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:11.160376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.160350    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1612dd-403c-4e3b-a679-23748bd36b1a-tls-certs\") pod \"kube-auth-proxy-65488ff6c7-b8fc4\" (UID: \"2a1612dd-403c-4e3b-a679-23748bd36b1a\") " pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:11.355582 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.355552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"
Sep 15 19:00:11.426721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.426690    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"]
Sep 15 19:00:11.434051 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.434024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.436372 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.436345    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 19:00:11.436583 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.436525    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 19:00:11.436583 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.436550    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 19:00:11.436756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.436734    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 19:00:11.436899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.436876    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jl9bt\""
Sep 15 19:00:11.441102 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.441081    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"]
Sep 15 19:00:11.561897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.561851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.562066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.561913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b322b894-053a-4e46-8f24-2c72a523cf69-cert\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.562066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.561938    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.562066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.561968    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.562066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.561993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.562066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.562051    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.562251 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.562070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61319e0-5299-4418-af8c-90126a5e4d21-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.562251 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.562093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zhdc\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-kube-api-access-4zhdc\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.564285 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.564264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b322b894-053a-4e46-8f24-2c72a523cf69-cert\") pod \"odh-model-controller-7998c67dc4-sxr2k\" (UID: \"b322b894-053a-4e46-8f24-2c72a523cf69\") " pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.631046 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.630972    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4"]
Sep 15 19:00:11.634517 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:11.634489    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1612dd_403c_4e3b_a679_23748bd36b1a.slice/crio-45c1ea480251b15a55947b2fec8d468cfd559c720bb98917a4e4504ddb919b84 WatchSource:0}: Error finding container 45c1ea480251b15a55947b2fec8d468cfd559c720bb98917a4e4504ddb919b84: Status 404 returned error can't find the container with id 45c1ea480251b15a55947b2fec8d468cfd559c720bb98917a4e4504ddb919b84
Sep 15 19:00:11.663263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.663368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.663368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.663368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663326    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.663505 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61319e0-5299-4418-af8c-90126a5e4d21-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.663542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zhdc\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-kube-api-access-4zhdc\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.663587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.663577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.664029 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.664002    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.666026 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.666004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.666118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.666048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.666118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.666084    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61319e0-5299-4418-af8c-90126a5e4d21-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.666542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.666523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.670982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.670960    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zhdc\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-kube-api-access-4zhdc\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.671047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.671031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vzp6r\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.728980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.728958    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:11.746078 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.746057    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:11.999002 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:11.998979    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sxr2k"]
Sep 15 19:00:12.000576 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:12.000554    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb322b894_053a_4e46_8f24_2c72a523cf69.slice/crio-5aeb8c90c3c04b71007009af3e4720acdfb35e29c11eaca1d55fa3ff9b78a1e0 WatchSource:0}: Error finding container 5aeb8c90c3c04b71007009af3e4720acdfb35e29c11eaca1d55fa3ff9b78a1e0: Status 404 returned error can't find the container with id 5aeb8c90c3c04b71007009af3e4720acdfb35e29c11eaca1d55fa3ff9b78a1e0
Sep 15 19:00:12.023137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:12.023118    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"]
Sep 15 19:00:12.024900 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:12.024881    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61319e0_5299_4418_af8c_90126a5e4d21.slice/crio-d8acf6c4fc6d06b0325762db59bcaf307481a6bd31c6408c38e5d5a984cee921 WatchSource:0}: Error finding container d8acf6c4fc6d06b0325762db59bcaf307481a6bd31c6408c38e5d5a984cee921: Status 404 returned error can't find the container with id d8acf6c4fc6d06b0325762db59bcaf307481a6bd31c6408c38e5d5a984cee921
Sep 15 19:00:12.252482 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:12.252385    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k" event={"ID":"b322b894-053a-4e46-8f24-2c72a523cf69","Type":"ContainerStarted","Data":"5aeb8c90c3c04b71007009af3e4720acdfb35e29c11eaca1d55fa3ff9b78a1e0"}
Sep 15 19:00:12.253661 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:12.253634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4" event={"ID":"2a1612dd-403c-4e3b-a679-23748bd36b1a","Type":"ContainerStarted","Data":"45c1ea480251b15a55947b2fec8d468cfd559c720bb98917a4e4504ddb919b84"}
Sep 15 19:00:12.254909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:12.254888    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" event={"ID":"b61319e0-5299-4418-af8c-90126a5e4d21","Type":"ContainerStarted","Data":"d8acf6c4fc6d06b0325762db59bcaf307481a6bd31c6408c38e5d5a984cee921"}
Sep 15 19:00:14.877043 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:14.877010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:17.730916 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:17.730855    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:00:17.731354 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:17.730942    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:00:18.294757 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.294711    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4" event={"ID":"2a1612dd-403c-4e3b-a679-23748bd36b1a","Type":"ContainerStarted","Data":"dd4efa2efbe5b37ee0c23683f380b0e40202c658e863439c041149ca63b334f7"}
Sep 15 19:00:18.296419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.296382    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" event={"ID":"b61319e0-5299-4418-af8c-90126a5e4d21","Type":"ContainerStarted","Data":"01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9"}
Sep 15 19:00:18.296561 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.296534    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:18.298008 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.297983    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k" event={"ID":"b322b894-053a-4e46-8f24-2c72a523cf69","Type":"ContainerStarted","Data":"30e6db595f7cf9952fce7a32ca07bee66ce6b267b0f94ff2bc576338cc936f90"}
Sep 15 19:00:18.298185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.298161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:18.298590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.298571    2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vzp6r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:00:18.298684 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.298631    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:00:18.311163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.311118    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65488ff6c7-b8fc4" podStartSLOduration=2.20614655 podStartE2EDuration="8.31110324s" totalImagesPullingTime="6.10495669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:18.309457397 +0000 UTC m=+570.463825865" watchObservedRunningTime="2026-09-15 19:00:18.31110324 +0000 UTC m=+570.465471756"
Sep 15 19:00:18.346788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.346734    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k" podStartSLOduration=2.6066763120000003 podStartE2EDuration="8.346712727s" totalImagesPullingTime="5.740036415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:18.326553713 +0000 UTC m=+570.480922183" watchObservedRunningTime="2026-09-15 19:00:18.346712727 +0000 UTC m=+570.501081196"
Sep 15 19:00:18.350239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.350191    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" podStartSLOduration=1.646527552 podStartE2EDuration="7.350172051s" totalImagesPullingTime="5.703644499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:18.348296884 +0000 UTC m=+570.502665345" watchObservedRunningTime="2026-09-15 19:00:18.350172051 +0000 UTC m=+570.504540520"
Sep 15 19:00:18.609956 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.609851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:18.709108 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:18.709062    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:19.099557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.099521    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"]
Sep 15 19:00:19.103588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.103562    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.106408 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.106383    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-n79ns\""
Sep 15 19:00:19.117362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.117327    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"]
Sep 15 19:00:19.133800 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzlc\" (UniqueName: \"kubernetes.io/projected/ad422d98-f7bb-4f31-9101-a1a9446fdb25-kube-api-access-7nzlc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.133980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.133980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.133980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.133980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.133980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133955    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.134158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.133986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.134158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.134045    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.134158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.134095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.137245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.137217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:19.234529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.234730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzlc\" (UniqueName: \"kubernetes.io/projected/ad422d98-f7bb-4f31-9101-a1a9446fdb25-kube-api-access-7nzlc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.234818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.234818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.234818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234785    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.234818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234816    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234878    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.234910    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.235187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.235196    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.235447    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235602 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.235576    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.235858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.235841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.237126 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.237101    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.237579 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.237559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.243281 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.243257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad422d98-f7bb-4f31-9101-a1a9446fdb25-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.243436 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.243415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzlc\" (UniqueName: \"kubernetes.io/projected/ad422d98-f7bb-4f31-9101-a1a9446fdb25-kube-api-access-7nzlc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62\" (UID: \"ad422d98-f7bb-4f31-9101-a1a9446fdb25\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.304188 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.304159    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:19.416846 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.416759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:19.428085 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.428057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:19.528117 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.528073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:19.699130 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:19.699103    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"]
Sep 15 19:00:19.701370 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:19.701342    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad422d98_f7bb_4f31_9101_a1a9446fdb25.slice/crio-1089a2ea752dce3b8541b98406141c759843b03f2e911ef47fec79d103b217cc WatchSource:0}: Error finding container 1089a2ea752dce3b8541b98406141c759843b03f2e911ef47fec79d103b217cc: Status 404 returned error can't find the container with id 1089a2ea752dce3b8541b98406141c759843b03f2e911ef47fec79d103b217cc
Sep 15 19:00:20.309605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.309571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62" event={"ID":"ad422d98-f7bb-4f31-9101-a1a9446fdb25","Type":"ContainerStarted","Data":"1089a2ea752dce3b8541b98406141c759843b03f2e911ef47fec79d103b217cc"}
Sep 15 19:00:20.797200 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.797145    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:20.835110 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.835074    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"]
Sep 15 19:00:20.838888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.838863    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:20.841797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.841407    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xfp9w\""
Sep 15 19:00:20.841797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.841636    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 19:00:20.861148 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.859753    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"]
Sep 15 19:00:20.863501 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.863454    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"]
Sep 15 19:00:20.863635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.863569    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:20.868652 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.867372    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nsm2t\""
Sep 15 19:00:20.868652 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.867579    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 19:00:20.878529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.878466    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"]
Sep 15 19:00:20.958134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.958089    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9w4\" (UniqueName: \"kubernetes.io/projected/2444b57d-2402-4b84-962b-5d06909775d8-kube-api-access-8b9w4\") pod \"llmisvc-controller-manager-777c6bd7d5-9g9t4\" (UID: \"2444b57d-2402-4b84-962b-5d06909775d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:20.958134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.958128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trnv\" (UniqueName: \"kubernetes.io/projected/dcaa3581-8da6-4a75-8ca4-b453c866e980-kube-api-access-7trnv\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:20.958362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.958158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcaa3581-8da6-4a75-8ca4-b453c866e980-cert\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:20.958362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:20.958189    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2444b57d-2402-4b84-962b-5d06909775d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9g9t4\" (UID: \"2444b57d-2402-4b84-962b-5d06909775d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:21.058821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.058722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2444b57d-2402-4b84-962b-5d06909775d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9g9t4\" (UID: \"2444b57d-2402-4b84-962b-5d06909775d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:21.059019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.058828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9w4\" (UniqueName: \"kubernetes.io/projected/2444b57d-2402-4b84-962b-5d06909775d8-kube-api-access-8b9w4\") pod \"llmisvc-controller-manager-777c6bd7d5-9g9t4\" (UID: \"2444b57d-2402-4b84-962b-5d06909775d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:21.059019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.058870    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trnv\" (UniqueName: \"kubernetes.io/projected/dcaa3581-8da6-4a75-8ca4-b453c866e980-kube-api-access-7trnv\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:21.059019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.058967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcaa3581-8da6-4a75-8ca4-b453c866e980-cert\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:21.059258 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:21.059121    2584 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 15 19:00:21.059258 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:21.059182    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcaa3581-8da6-4a75-8ca4-b453c866e980-cert podName:dcaa3581-8da6-4a75-8ca4-b453c866e980 nodeName:}" failed. No retries permitted until 2026-09-15 19:00:21.559161464 +0000 UTC m=+573.713529919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dcaa3581-8da6-4a75-8ca4-b453c866e980-cert") pod "kserve-controller-manager-d6f4b65b8-h5m64" (UID: "dcaa3581-8da6-4a75-8ca4-b453c866e980") : secret "kserve-webhook-server-cert" not found
Sep 15 19:00:21.062014 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.061987    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2444b57d-2402-4b84-962b-5d06909775d8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9g9t4\" (UID: \"2444b57d-2402-4b84-962b-5d06909775d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:21.085282 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.085233    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9w4\" (UniqueName: \"kubernetes.io/projected/2444b57d-2402-4b84-962b-5d06909775d8-kube-api-access-8b9w4\") pod \"llmisvc-controller-manager-777c6bd7d5-9g9t4\" (UID: \"2444b57d-2402-4b84-962b-5d06909775d8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:21.085418 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.085385    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trnv\" (UniqueName: \"kubernetes.io/projected/dcaa3581-8da6-4a75-8ca4-b453c866e980-kube-api-access-7trnv\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:21.185778 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.185521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:21.405379 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.405287    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:21.470703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.470679    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"]
Sep 15 19:00:21.472878 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:21.472852    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2444b57d_2402_4b84_962b_5d06909775d8.slice/crio-75c84dceb6f4fcbdb9566c7ce9fd1ea05d25ed77ecaa23763dc851680f690786 WatchSource:0}: Error finding container 75c84dceb6f4fcbdb9566c7ce9fd1ea05d25ed77ecaa23763dc851680f690786: Status 404 returned error can't find the container with id 75c84dceb6f4fcbdb9566c7ce9fd1ea05d25ed77ecaa23763dc851680f690786
Sep 15 19:00:21.474272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.474242    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:21.563311 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.563272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcaa3581-8da6-4a75-8ca4-b453c866e980-cert\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:21.566076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.566050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcaa3581-8da6-4a75-8ca4-b453c866e980-cert\") pod \"kserve-controller-manager-d6f4b65b8-h5m64\" (UID: \"dcaa3581-8da6-4a75-8ca4-b453c866e980\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:21.765788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:21.765753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:22.192396 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.192318    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"]
Sep 15 19:00:22.192537 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.192515    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" containerName="discovery" containerID="cri-o://01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9" gracePeriod=30
Sep 15 19:00:22.304301 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.304262    2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vzp6r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:8080/ready\": dial tcp 10.133.0.29:8080: connect: connection refused" start-of-body=
Sep 15 19:00:22.304463 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.304329    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.29:8080/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 15 19:00:22.321936 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.321903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4" event={"ID":"2444b57d-2402-4b84-962b-5d06909775d8","Type":"ContainerStarted","Data":"75c84dceb6f4fcbdb9566c7ce9fd1ea05d25ed77ecaa23763dc851680f690786"}
Sep 15 19:00:22.693901 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.693863    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:00:22.694272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.693932    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:00:22.694272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.693960    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:00:22.773980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.773958    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:22.877550 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877470    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zhdc\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-kube-api-access-4zhdc\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.877550 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877526    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61319e0-5299-4418-af8c-90126a5e4d21-local-certs\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.877799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877570    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-ca-configmap\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.877799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-cacerts\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.877799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877604    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-dns-cert\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.877799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877639    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-istio-token\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.877799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877669    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-kubeconfig\") pod \"b61319e0-5299-4418-af8c-90126a5e4d21\" (UID: \"b61319e0-5299-4418-af8c-90126a5e4d21\") "
Sep 15 19:00:22.878063 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.877997    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:00:22.880337 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.880292    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:00:22.880337 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.880323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61319e0-5299-4418-af8c-90126a5e4d21-local-certs" (OuterVolumeSpecName: "local-certs") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:00:22.880533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.880447    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:00:22.880533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.880526    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-cacerts" (OuterVolumeSpecName: "cacerts") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:00:22.881182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.881151    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-istio-token" (OuterVolumeSpecName: "istio-token") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:00:22.881436 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.881412    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-kube-api-access-4zhdc" (OuterVolumeSpecName: "kube-api-access-4zhdc") pod "b61319e0-5299-4418-af8c-90126a5e4d21" (UID: "b61319e0-5299-4418-af8c-90126a5e4d21"). InnerVolumeSpecName "kube-api-access-4zhdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:00:22.891954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.891781    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"]
Sep 15 19:00:22.893938 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:22.893914    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaa3581_8da6_4a75_8ca4_b453c866e980.slice/crio-aade11ad7ee57c6e98c990f8a362755e0c8e15445f62c91cbd84337401d12970 WatchSource:0}: Error finding container aade11ad7ee57c6e98c990f8a362755e0c8e15445f62c91cbd84337401d12970: Status 404 returned error can't find the container with id aade11ad7ee57c6e98c990f8a362755e0c8e15445f62c91cbd84337401d12970
Sep 15 19:00:22.979239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979212    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-ca-configmap\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:22.979239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979236    2584 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-cacerts\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:22.979239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979247    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-csr-dns-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:22.979450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979257    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:22.979450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979265    2584 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61319e0-5299-4418-af8c-90126a5e4d21-istio-kubeconfig\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:22.979450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979273    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zhdc\" (UniqueName: \"kubernetes.io/projected/b61319e0-5299-4418-af8c-90126a5e4d21-kube-api-access-4zhdc\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:22.979450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:22.979284    2584 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61319e0-5299-4418-af8c-90126a5e4d21-local-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:00:23.328974 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.328941    2584 generic.go:356] "Generic (PLEG): container finished" podID="b61319e0-5299-4418-af8c-90126a5e4d21" containerID="01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9" exitCode=0
Sep 15 19:00:23.329162 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.329005    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"
Sep 15 19:00:23.329162 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.329004    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" event={"ID":"b61319e0-5299-4418-af8c-90126a5e4d21","Type":"ContainerDied","Data":"01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9"}
Sep 15 19:00:23.329162 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.329072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r" event={"ID":"b61319e0-5299-4418-af8c-90126a5e4d21","Type":"ContainerDied","Data":"d8acf6c4fc6d06b0325762db59bcaf307481a6bd31c6408c38e5d5a984cee921"}
Sep 15 19:00:23.329162 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.329101    2584 scope.go:117] "RemoveContainer" containerID="01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9"
Sep 15 19:00:23.331189 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.331118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62" event={"ID":"ad422d98-f7bb-4f31-9101-a1a9446fdb25","Type":"ContainerStarted","Data":"d6865f2815acf6ba42b4c8cf8a1eb2d04dc097394977fd368af1a8c255ff8d38"}
Sep 15 19:00:23.332767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.332742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64" event={"ID":"dcaa3581-8da6-4a75-8ca4-b453c866e980","Type":"ContainerStarted","Data":"aade11ad7ee57c6e98c990f8a362755e0c8e15445f62c91cbd84337401d12970"}
Sep 15 19:00:23.344789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.340235    2584 scope.go:117] "RemoveContainer" containerID="01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9"
Sep 15 19:00:23.345250 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:00:23.345222    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9\": container with ID starting with 01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9 not found: ID does not exist" containerID="01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9"
Sep 15 19:00:23.345399 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.345367    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9"} err="failed to get container status \"01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9\": rpc error: code = NotFound desc = could not find container \"01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9\": container with ID starting with 01831eef6338337bd5d053fd448558e199161cd837a770d396db8578526ce8b9 not found: ID does not exist"
Sep 15 19:00:23.353333 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.353295    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62" podStartSLOduration=1.363668256 podStartE2EDuration="4.353283849s" totalImagesPullingTime="2.989615593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:23.352488371 +0000 UTC m=+575.506856839" watchObservedRunningTime="2026-09-15 19:00:23.353283849 +0000 UTC m=+575.507652317"
Sep 15 19:00:23.367641 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.367599    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"]
Sep 15 19:00:23.371959 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.371939    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vzp6r"]
Sep 15 19:00:23.416960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.416935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:23.421774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:23.421757    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:24.234653 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:24.234622    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" path="/var/lib/kubelet/pods/b61319e0-5299-4418-af8c-90126a5e4d21/volumes"
Sep 15 19:00:24.339121 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:24.339095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:24.340152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:24.340122    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62"
Sep 15 19:00:26.350074 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:26.350042    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4" event={"ID":"2444b57d-2402-4b84-962b-5d06909775d8","Type":"ContainerStarted","Data":"432fcd8230be411dd295f0fb73dcb88013e3b9a950ab68398e403b29a35277ea"}
Sep 15 19:00:26.350413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:26.350104    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:26.351672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:26.351639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64" event={"ID":"dcaa3581-8da6-4a75-8ca4-b453c866e980","Type":"ContainerStarted","Data":"9416163cfd9d6da799d56556b7f2d36b8c93912b76ca6f48fb8e2a8e5dfeb2f5"}
Sep 15 19:00:26.351916 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:26.351899    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:00:26.366188 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:26.366148    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4" podStartSLOduration=1.59337792 podStartE2EDuration="6.366133836s" totalImagesPullingTime="4.772755916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:26.363604006 +0000 UTC m=+578.517972476" watchObservedRunningTime="2026-09-15 19:00:26.366133836 +0000 UTC m=+578.520502304"
Sep 15 19:00:26.378434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:26.378398    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64" podStartSLOduration=3.839772028 podStartE2EDuration="6.378386059s" totalImagesPullingTime="2.538614031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:26.378027864 +0000 UTC m=+578.532396333" watchObservedRunningTime="2026-09-15 19:00:26.378386059 +0000 UTC m=+578.532754527"
Sep 15 19:00:29.305645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.305605    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sxr2k"
Sep 15 19:00:29.776342 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.776310    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64f57b995d-m9kwk"]
Sep 15 19:00:29.776672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.776642    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" containerName="discovery"
Sep 15 19:00:29.776672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.776658    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" containerName="discovery"
Sep 15 19:00:29.776776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.776719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61319e0-5299-4418-af8c-90126a5e4d21" containerName="discovery"
Sep 15 19:00:29.779830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.779811    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.783137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783044    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 19:00:29.783137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783075    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 19:00:29.783137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783086    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 19:00:29.783341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783141    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 19:00:29.783341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783044    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 19:00:29.783497 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783479    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 19:00:29.783497 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783489    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 19:00:29.783696 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.783600    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kbhdv\""
Sep 15 19:00:29.786730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.786712    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 19:00:29.790264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.790245    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f57b995d-m9kwk"]
Sep 15 19:00:29.830128 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.830103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:29.842617 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-oauth-serving-cert\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.842680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mpk\" (UniqueName: \"kubernetes.io/projected/a3d86932-ad31-42cd-b23c-fead3b3f07e6-kube-api-access-26mpk\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.842728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-serving-cert\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.842764 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-config\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.842798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-service-ca\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.842798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-trusted-ca-bundle\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.842860 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.842847    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-oauth-config\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944136 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944107    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-config\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944136 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944143    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-service-ca\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-trusted-ca-bundle\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-oauth-config\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944254    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-oauth-serving-cert\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944279    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mpk\" (UniqueName: \"kubernetes.io/projected/a3d86932-ad31-42cd-b23c-fead3b3f07e6-kube-api-access-26mpk\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.944572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-serving-cert\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.945032 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944915    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-config\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.945032 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.944989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-service-ca\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.945220 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.945100    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-oauth-serving-cert\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.945220 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.945100    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3d86932-ad31-42cd-b23c-fead3b3f07e6-trusted-ca-bundle\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.947097 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.947068    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-oauth-config\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.947181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.947139    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3d86932-ad31-42cd-b23c-fead3b3f07e6-console-serving-cert\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:29.951480 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:29.951457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mpk\" (UniqueName: \"kubernetes.io/projected/a3d86932-ad31-42cd-b23c-fead3b3f07e6-kube-api-access-26mpk\") pod \"console-64f57b995d-m9kwk\" (UID: \"a3d86932-ad31-42cd-b23c-fead3b3f07e6\") " pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:30.089747 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:30.089661    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:30.380500 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:00:30.380460    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d86932_ad31_42cd_b23c_fead3b3f07e6.slice/crio-6d514903a3664ee255e0fd1a148b075170475accf1b9b15fac309a130b458fa8 WatchSource:0}: Error finding container 6d514903a3664ee255e0fd1a148b075170475accf1b9b15fac309a130b458fa8: Status 404 returned error can't find the container with id 6d514903a3664ee255e0fd1a148b075170475accf1b9b15fac309a130b458fa8
Sep 15 19:00:30.381691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:30.381668    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64f57b995d-m9kwk"]
Sep 15 19:00:31.381254 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:31.381219    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f57b995d-m9kwk" event={"ID":"a3d86932-ad31-42cd-b23c-fead3b3f07e6","Type":"ContainerStarted","Data":"fee522cd283c4f16e2e055e4b226cfc07e5d48555276a8df781fb075c288ca33"}
Sep 15 19:00:31.381762 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:31.381272    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64f57b995d-m9kwk" event={"ID":"a3d86932-ad31-42cd-b23c-fead3b3f07e6","Type":"ContainerStarted","Data":"6d514903a3664ee255e0fd1a148b075170475accf1b9b15fac309a130b458fa8"}
Sep 15 19:00:31.397136 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:31.397081    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64f57b995d-m9kwk" podStartSLOduration=2.397065323 podStartE2EDuration="2.397065323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:00:31.395757023 +0000 UTC m=+583.550125491" watchObservedRunningTime="2026-09-15 19:00:31.397065323 +0000 UTC m=+583.551433790"
Sep 15 19:00:40.089970 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.089935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:40.090347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.089994    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:40.094503 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.094479    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:40.428419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.428337    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64f57b995d-m9kwk"
Sep 15 19:00:40.583287 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.583257    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:40.640651 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.640603    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:40.718388 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:40.718331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:45.248351 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:45.248321    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:50.514944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:50.514908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:00:57.358435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:57.358405    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9g9t4"
Sep 15 19:00:57.362047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:00:57.362023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-h5m64"
Sep 15 19:01:06.433037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:06.433005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:11.749940 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:11.749864    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:11.852543 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:11.852491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:12.224770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:12.224739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:12.324691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:12.324642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:14.158467 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:14.158436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:15.312754 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.312722    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"]
Sep 15 19:01:15.315312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.315296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.317976 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.317955    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 19:01:15.318106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.317994    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 19:01:15.318676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.318652    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nljx\""
Sep 15 19:01:15.326859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.326833    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"]
Sep 15 19:01:15.410692 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.410643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.410870 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.410738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hftm\" (UniqueName: \"kubernetes.io/projected/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-kube-api-access-6hftm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.410870 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.410775    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.412716 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.412692    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"]
Sep 15 19:01:15.415548 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.415530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.429628 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.426222    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"]
Sep 15 19:01:15.511862 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.511830    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.511998 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.511877    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.512042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.511993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.512042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.512025    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dtt\" (UniqueName: \"kubernetes.io/projected/72eeca64-93f1-4f48-8448-f301ee14c4ff-kube-api-access-f8dtt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.512183 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.512162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hftm\" (UniqueName: \"kubernetes.io/projected/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-kube-api-access-6hftm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.512246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.512195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.512246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.512209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.512387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.512373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.512449 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.512431    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"]
Sep 15 19:01:15.514922 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.514908    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.520355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.520336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hftm\" (UniqueName: \"kubernetes.io/projected/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-kube-api-access-6hftm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.525165 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.525143    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"]
Sep 15 19:01:15.613006 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.612927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.613006 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.612960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dtt\" (UniqueName: \"kubernetes.io/projected/72eeca64-93f1-4f48-8448-f301ee14c4ff-kube-api-access-f8dtt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.613006 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.612989    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krv82\" (UniqueName: \"kubernetes.io/projected/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-kube-api-access-krv82\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.613267 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.613017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.613267 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.613142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.613267 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.613167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.613422 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.613368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.613460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.613423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.615829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.615808    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"]
Sep 15 19:01:15.618109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.618095    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.622538 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.622517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dtt\" (UniqueName: \"kubernetes.io/projected/72eeca64-93f1-4f48-8448-f301ee14c4ff-kube-api-access-f8dtt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.624177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.624160    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:15.628556 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.628536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"]
Sep 15 19:01:15.714521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krv82\" (UniqueName: \"kubernetes.io/projected/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-kube-api-access-krv82\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.714665 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.714665 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714554    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.714665 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbq5t\" (UniqueName: \"kubernetes.io/projected/20e8ee40-3f49-4960-8609-bb79e3e1db85-kube-api-access-vbq5t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.714665 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.714798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.714968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.714949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.715040 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.715017    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.722118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.722093    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krv82\" (UniqueName: \"kubernetes.io/projected/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-kube-api-access-krv82\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.724915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.724898    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:15.815891 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.815862    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.815993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.815911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.815993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.815936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbq5t\" (UniqueName: \"kubernetes.io/projected/20e8ee40-3f49-4960-8609-bb79e3e1db85-kube-api-access-vbq5t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.816223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.816205    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.816302 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.816286    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.825334 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.825305    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbq5t\" (UniqueName: \"kubernetes.io/projected/20e8ee40-3f49-4960-8609-bb79e3e1db85-kube-api-access-vbq5t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.833154 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.833134    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:15.897120 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.897096    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"]
Sep 15 19:01:15.898887 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:15.898862    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100ea90d_c077_4bf8_a5ec_ba39e9ec856e.slice/crio-c30c26cac0e26f05ffa93adccc128514e82ccd470bb7e775cefed5e5534f7027 WatchSource:0}: Error finding container c30c26cac0e26f05ffa93adccc128514e82ccd470bb7e775cefed5e5534f7027: Status 404 returned error can't find the container with id c30c26cac0e26f05ffa93adccc128514e82ccd470bb7e775cefed5e5534f7027
Sep 15 19:01:15.948265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.948243    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:15.992295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:15.992270    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"]
Sep 15 19:01:16.103304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.103279    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"]
Sep 15 19:01:16.105170 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:16.105145    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e07fa3_47ae_4ec6_8af0_417a3a59f97f.slice/crio-a251192756a181137048f389ff4fcfed0f25a43f626bb2afa4f4dbcb23646cf1 WatchSource:0}: Error finding container a251192756a181137048f389ff4fcfed0f25a43f626bb2afa4f4dbcb23646cf1: Status 404 returned error can't find the container with id a251192756a181137048f389ff4fcfed0f25a43f626bb2afa4f4dbcb23646cf1
Sep 15 19:01:16.219403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.219377    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"]
Sep 15 19:01:16.221878 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:16.221855    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e8ee40_3f49_4960_8609_bb79e3e1db85.slice/crio-3f6180b86906ead4c1bcd2838b4411c133bb7a354f6b17e8c9b2543616032a22 WatchSource:0}: Error finding container 3f6180b86906ead4c1bcd2838b4411c133bb7a354f6b17e8c9b2543616032a22: Status 404 returned error can't find the container with id 3f6180b86906ead4c1bcd2838b4411c133bb7a354f6b17e8c9b2543616032a22
Sep 15 19:01:16.592202 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.592171    2584 generic.go:356] "Generic (PLEG): container finished" podID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerID="9893496a2acb573f774ef82afa66e6a88706c9b8d7d1502c17807e56bb46f35a" exitCode=0
Sep 15 19:01:16.592594 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.592263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb" event={"ID":"72eeca64-93f1-4f48-8448-f301ee14c4ff","Type":"ContainerDied","Data":"9893496a2acb573f774ef82afa66e6a88706c9b8d7d1502c17807e56bb46f35a"}
Sep 15 19:01:16.592594 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.592323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb" event={"ID":"72eeca64-93f1-4f48-8448-f301ee14c4ff","Type":"ContainerStarted","Data":"4613a7e1afbcfa63e99a1feec2c686013432b8cd77f11ad48d9c8390b3d958b1"}
Sep 15 19:01:16.593893 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.593873    2584 generic.go:356] "Generic (PLEG): container finished" podID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerID="70524b3384f4c3288b8fb3f210b0dd6b6d4950cde2990bd43ed3e44842b89984" exitCode=0
Sep 15 19:01:16.593982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.593967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575" event={"ID":"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f","Type":"ContainerDied","Data":"70524b3384f4c3288b8fb3f210b0dd6b6d4950cde2990bd43ed3e44842b89984"}
Sep 15 19:01:16.594047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.594007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575" event={"ID":"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f","Type":"ContainerStarted","Data":"a251192756a181137048f389ff4fcfed0f25a43f626bb2afa4f4dbcb23646cf1"}
Sep 15 19:01:16.595744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.595721    2584 generic.go:356] "Generic (PLEG): container finished" podID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerID="78b1585989f3a10b579a2913a002d301740d9b264117261a7d1f76a652f40911" exitCode=0
Sep 15 19:01:16.595853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.595810    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f" event={"ID":"20e8ee40-3f49-4960-8609-bb79e3e1db85","Type":"ContainerDied","Data":"78b1585989f3a10b579a2913a002d301740d9b264117261a7d1f76a652f40911"}
Sep 15 19:01:16.595927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.595866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f" event={"ID":"20e8ee40-3f49-4960-8609-bb79e3e1db85","Type":"ContainerStarted","Data":"3f6180b86906ead4c1bcd2838b4411c133bb7a354f6b17e8c9b2543616032a22"}
Sep 15 19:01:16.597763 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.597679    2584 generic.go:356] "Generic (PLEG): container finished" podID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerID="b19f8c2f37862eaef5112cb45824ede1261c9f906dff9093c5fb0c7065c29d2b" exitCode=0
Sep 15 19:01:16.597763 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.597718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj" event={"ID":"100ea90d-c077-4bf8-a5ec-ba39e9ec856e","Type":"ContainerDied","Data":"b19f8c2f37862eaef5112cb45824ede1261c9f906dff9093c5fb0c7065c29d2b"}
Sep 15 19:01:16.597763 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:16.597759    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj" event={"ID":"100ea90d-c077-4bf8-a5ec-ba39e9ec856e","Type":"ContainerStarted","Data":"c30c26cac0e26f05ffa93adccc128514e82ccd470bb7e775cefed5e5534f7027"}
Sep 15 19:01:17.610117 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:17.610083    2584 generic.go:356] "Generic (PLEG): container finished" podID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerID="a60ed2c735b620058281961897aceb293b3cdd3aecd27577c56e0e9dc7a95831" exitCode=0
Sep 15 19:01:17.610536 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:17.610161    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj" event={"ID":"100ea90d-c077-4bf8-a5ec-ba39e9ec856e","Type":"ContainerDied","Data":"a60ed2c735b620058281961897aceb293b3cdd3aecd27577c56e0e9dc7a95831"}
Sep 15 19:01:18.617193 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:18.617159    2584 generic.go:356] "Generic (PLEG): container finished" podID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerID="4d340f0f6da8ccce432da6ba06123a594a1c919efe542af17f93fda617dfa3c8" exitCode=0
Sep 15 19:01:18.617586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:18.617241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj" event={"ID":"100ea90d-c077-4bf8-a5ec-ba39e9ec856e","Type":"ContainerDied","Data":"4d340f0f6da8ccce432da6ba06123a594a1c919efe542af17f93fda617dfa3c8"}
Sep 15 19:01:18.619021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:18.618999    2584 generic.go:356] "Generic (PLEG): container finished" podID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerID="ab64f0c1b2e9e26ff61970ab1fe7032cc84a3f85e0f39ff46c32d3c6d2fcc38d" exitCode=0
Sep 15 19:01:18.619155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:18.619037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb" event={"ID":"72eeca64-93f1-4f48-8448-f301ee14c4ff","Type":"ContainerDied","Data":"ab64f0c1b2e9e26ff61970ab1fe7032cc84a3f85e0f39ff46c32d3c6d2fcc38d"}
Sep 15 19:01:19.628199 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.628167    2584 generic.go:356] "Generic (PLEG): container finished" podID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerID="0bd2ce5bca47c9db1af5979ad6348739756629b8ca10d2203b5e4678df441ffe" exitCode=0
Sep 15 19:01:19.628625 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.628255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb" event={"ID":"72eeca64-93f1-4f48-8448-f301ee14c4ff","Type":"ContainerDied","Data":"0bd2ce5bca47c9db1af5979ad6348739756629b8ca10d2203b5e4678df441ffe"}
Sep 15 19:01:19.794116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.794089    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:19.851913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.851891    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-util\") pod \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") "
Sep 15 19:01:19.852010 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.851946    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hftm\" (UniqueName: \"kubernetes.io/projected/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-kube-api-access-6hftm\") pod \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") "
Sep 15 19:01:19.852010 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.851981    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-bundle\") pod \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\" (UID: \"100ea90d-c077-4bf8-a5ec-ba39e9ec856e\") "
Sep 15 19:01:19.852460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.852431    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-bundle" (OuterVolumeSpecName: "bundle") pod "100ea90d-c077-4bf8-a5ec-ba39e9ec856e" (UID: "100ea90d-c077-4bf8-a5ec-ba39e9ec856e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:19.853753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.853733    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-kube-api-access-6hftm" (OuterVolumeSpecName: "kube-api-access-6hftm") pod "100ea90d-c077-4bf8-a5ec-ba39e9ec856e" (UID: "100ea90d-c077-4bf8-a5ec-ba39e9ec856e"). InnerVolumeSpecName "kube-api-access-6hftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:01:19.857749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.857716    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-util" (OuterVolumeSpecName: "util") pod "100ea90d-c077-4bf8-a5ec-ba39e9ec856e" (UID: "100ea90d-c077-4bf8-a5ec-ba39e9ec856e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:19.953391 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.953365    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:19.953391 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.953389    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hftm\" (UniqueName: \"kubernetes.io/projected/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-kube-api-access-6hftm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:19.953516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:19.953399    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/100ea90d-c077-4bf8-a5ec-ba39e9ec856e-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:20.638470 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.638425    2584 generic.go:356] "Generic (PLEG): container finished" podID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerID="e542d7b550ff9b70251cadd363139de03c45ea3911b775300a58757cb8ea15e3" exitCode=0
Sep 15 19:01:20.638852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.638500    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575" event={"ID":"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f","Type":"ContainerDied","Data":"e542d7b550ff9b70251cadd363139de03c45ea3911b775300a58757cb8ea15e3"}
Sep 15 19:01:20.640768 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.640749    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj"
Sep 15 19:01:20.640868 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.640767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6txnqj" event={"ID":"100ea90d-c077-4bf8-a5ec-ba39e9ec856e","Type":"ContainerDied","Data":"c30c26cac0e26f05ffa93adccc128514e82ccd470bb7e775cefed5e5534f7027"}
Sep 15 19:01:20.640868 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.640806    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30c26cac0e26f05ffa93adccc128514e82ccd470bb7e775cefed5e5534f7027"
Sep 15 19:01:20.835274 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.835247    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:20.963179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.963078    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-util\") pod \"72eeca64-93f1-4f48-8448-f301ee14c4ff\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") "
Sep 15 19:01:20.963179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.963135    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-bundle\") pod \"72eeca64-93f1-4f48-8448-f301ee14c4ff\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") "
Sep 15 19:01:20.963179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.963190    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8dtt\" (UniqueName: \"kubernetes.io/projected/72eeca64-93f1-4f48-8448-f301ee14c4ff-kube-api-access-f8dtt\") pod \"72eeca64-93f1-4f48-8448-f301ee14c4ff\" (UID: \"72eeca64-93f1-4f48-8448-f301ee14c4ff\") "
Sep 15 19:01:20.963709 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.963666    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-bundle" (OuterVolumeSpecName: "bundle") pod "72eeca64-93f1-4f48-8448-f301ee14c4ff" (UID: "72eeca64-93f1-4f48-8448-f301ee14c4ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:20.965389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.965362    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eeca64-93f1-4f48-8448-f301ee14c4ff-kube-api-access-f8dtt" (OuterVolumeSpecName: "kube-api-access-f8dtt") pod "72eeca64-93f1-4f48-8448-f301ee14c4ff" (UID: "72eeca64-93f1-4f48-8448-f301ee14c4ff"). InnerVolumeSpecName "kube-api-access-f8dtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:01:20.968504 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:20.968465    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-util" (OuterVolumeSpecName: "util") pod "72eeca64-93f1-4f48-8448-f301ee14c4ff" (UID: "72eeca64-93f1-4f48-8448-f301ee14c4ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:21.063979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.063942    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:21.063979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.063974    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72eeca64-93f1-4f48-8448-f301ee14c4ff-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:21.063979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.063983    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dtt\" (UniqueName: \"kubernetes.io/projected/72eeca64-93f1-4f48-8448-f301ee14c4ff-kube-api-access-f8dtt\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:21.648018 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.647991    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb"
Sep 15 19:01:21.648517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.648025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm45zb" event={"ID":"72eeca64-93f1-4f48-8448-f301ee14c4ff","Type":"ContainerDied","Data":"4613a7e1afbcfa63e99a1feec2c686013432b8cd77f11ad48d9c8390b3d958b1"}
Sep 15 19:01:21.648517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.648077    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4613a7e1afbcfa63e99a1feec2c686013432b8cd77f11ad48d9c8390b3d958b1"
Sep 15 19:01:21.650181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.650151    2584 generic.go:356] "Generic (PLEG): container finished" podID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerID="6d20c66b2044fa0daffe48e2f10a8978021aece5026b0e9e7068aa5ee0eb71fa" exitCode=0
Sep 15 19:01:21.650307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.650237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575" event={"ID":"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f","Type":"ContainerDied","Data":"6d20c66b2044fa0daffe48e2f10a8978021aece5026b0e9e7068aa5ee0eb71fa"}
Sep 15 19:01:21.651995 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.651973    2584 generic.go:356] "Generic (PLEG): container finished" podID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerID="c6cc26dfe98f6b589ff0f8f927359a5d08a9b9d822cac0408683cd4ff46293fe" exitCode=0
Sep 15 19:01:21.652118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:21.652029    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f" event={"ID":"20e8ee40-3f49-4960-8609-bb79e3e1db85","Type":"ContainerDied","Data":"c6cc26dfe98f6b589ff0f8f927359a5d08a9b9d822cac0408683cd4ff46293fe"}
Sep 15 19:01:22.660098 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.660068    2584 generic.go:356] "Generic (PLEG): container finished" podID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerID="6e18e2001bbf778455383bcfe8261999d604cfcd1d5c59fe0ea722f005b40815" exitCode=0
Sep 15 19:01:22.660508 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.660159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f" event={"ID":"20e8ee40-3f49-4960-8609-bb79e3e1db85","Type":"ContainerDied","Data":"6e18e2001bbf778455383bcfe8261999d604cfcd1d5c59fe0ea722f005b40815"}
Sep 15 19:01:22.788062 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.788036    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:22.878034 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.878007    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-util\") pod \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") "
Sep 15 19:01:22.878173 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.878041    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-bundle\") pod \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") "
Sep 15 19:01:22.878173 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.878078    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krv82\" (UniqueName: \"kubernetes.io/projected/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-kube-api-access-krv82\") pod \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\" (UID: \"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f\") "
Sep 15 19:01:22.878590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.878550    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-bundle" (OuterVolumeSpecName: "bundle") pod "a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" (UID: "a4e07fa3-47ae-4ec6-8af0-417a3a59f97f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:22.880146 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.880118    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-kube-api-access-krv82" (OuterVolumeSpecName: "kube-api-access-krv82") pod "a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" (UID: "a4e07fa3-47ae-4ec6-8af0-417a3a59f97f"). InnerVolumeSpecName "kube-api-access-krv82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:01:22.882719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.882699    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-util" (OuterVolumeSpecName: "util") pod "a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" (UID: "a4e07fa3-47ae-4ec6-8af0-417a3a59f97f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:22.979630 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.979540    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krv82\" (UniqueName: \"kubernetes.io/projected/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-kube-api-access-krv82\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:22.979630 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.979565    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:22.979630 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:22.979575    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4e07fa3-47ae-4ec6-8af0-417a3a59f97f-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:23.666789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.666755    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575"
Sep 15 19:01:23.666789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.666779    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cp575" event={"ID":"a4e07fa3-47ae-4ec6-8af0-417a3a59f97f","Type":"ContainerDied","Data":"a251192756a181137048f389ff4fcfed0f25a43f626bb2afa4f4dbcb23646cf1"}
Sep 15 19:01:23.667231 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.666832    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a251192756a181137048f389ff4fcfed0f25a43f626bb2afa4f4dbcb23646cf1"
Sep 15 19:01:23.789265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.789242    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:23.887749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.887707    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-bundle\") pod \"20e8ee40-3f49-4960-8609-bb79e3e1db85\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") "
Sep 15 19:01:23.887749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.887764    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-util\") pod \"20e8ee40-3f49-4960-8609-bb79e3e1db85\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") "
Sep 15 19:01:23.887965 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.887813    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbq5t\" (UniqueName: \"kubernetes.io/projected/20e8ee40-3f49-4960-8609-bb79e3e1db85-kube-api-access-vbq5t\") pod \"20e8ee40-3f49-4960-8609-bb79e3e1db85\" (UID: \"20e8ee40-3f49-4960-8609-bb79e3e1db85\") "
Sep 15 19:01:23.888355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.888330    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-bundle" (OuterVolumeSpecName: "bundle") pod "20e8ee40-3f49-4960-8609-bb79e3e1db85" (UID: "20e8ee40-3f49-4960-8609-bb79e3e1db85"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:23.889926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.889904    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e8ee40-3f49-4960-8609-bb79e3e1db85-kube-api-access-vbq5t" (OuterVolumeSpecName: "kube-api-access-vbq5t") pod "20e8ee40-3f49-4960-8609-bb79e3e1db85" (UID: "20e8ee40-3f49-4960-8609-bb79e3e1db85"). InnerVolumeSpecName "kube-api-access-vbq5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:01:23.891809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.891775    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-util" (OuterVolumeSpecName: "util") pod "20e8ee40-3f49-4960-8609-bb79e3e1db85" (UID: "20e8ee40-3f49-4960-8609-bb79e3e1db85"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:01:23.988977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.988900    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbq5t\" (UniqueName: \"kubernetes.io/projected/20e8ee40-3f49-4960-8609-bb79e3e1db85-kube-api-access-vbq5t\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:23.988977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.988927    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-bundle\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:23.988977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:23.988936    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20e8ee40-3f49-4960-8609-bb79e3e1db85-util\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:01:24.673185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:24.673141    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f" event={"ID":"20e8ee40-3f49-4960-8609-bb79e3e1db85","Type":"ContainerDied","Data":"3f6180b86906ead4c1bcd2838b4411c133bb7a354f6b17e8c9b2543616032a22"}
Sep 15 19:01:24.673185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:24.673181    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9r6f"
Sep 15 19:01:24.673560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:24.673197    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6180b86906ead4c1bcd2838b4411c133bb7a354f6b17e8c9b2543616032a22"
Sep 15 19:01:30.432574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:30.432540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:32.488159 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488124    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"]
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488459    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="pull"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488471    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="pull"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488484    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="util"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488492    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="util"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488500    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="util"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488506    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="util"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488513    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="extract"
Sep 15 19:01:32.488517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488519    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488526    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="pull"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488531    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="pull"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488539    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488544    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488553    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488560    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488570    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="util"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488576    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="util"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488581    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="util"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488585    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="util"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488592    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="pull"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488599    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="pull"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488629    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488635    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488647    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="pull"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488654    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="pull"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488709    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="72eeca64-93f1-4f48-8448-f301ee14c4ff" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="100ea90d-c077-4bf8-a5ec-ba39e9ec856e" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488730    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e07fa3-47ae-4ec6-8af0-417a3a59f97f" containerName="extract"
Sep 15 19:01:32.488770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.488736    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e8ee40-3f49-4960-8609-bb79e3e1db85" containerName="extract"
Sep 15 19:01:32.493473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.493457    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:32.496118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.496096    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jzx87\""
Sep 15 19:01:32.496867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.496845    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 19:01:32.496989 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.496852    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 19:01:32.505360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.505337    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"]
Sep 15 19:01:32.562440 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.562407    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzt96\" (UniqueName: \"kubernetes.io/projected/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c-kube-api-access-xzt96\") pod \"limitador-operator-controller-manager-74495c69f8-mg9b4\" (UID: \"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:32.663856 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.663817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzt96\" (UniqueName: \"kubernetes.io/projected/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c-kube-api-access-xzt96\") pod \"limitador-operator-controller-manager-74495c69f8-mg9b4\" (UID: \"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:32.672050 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.672026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzt96\" (UniqueName: \"kubernetes.io/projected/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c-kube-api-access-xzt96\") pod \"limitador-operator-controller-manager-74495c69f8-mg9b4\" (UID: \"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:32.803516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:32.803482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:33.078592 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:33.078568    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"]
Sep 15 19:01:33.080432 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:33.080405    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebd3ffa_6ac2_4cbd_b825_2b4e89569d0c.slice/crio-5c86154db19f4bb5807909548dd01dd6be9da8793248d43df03ae21bf88c57f8 WatchSource:0}: Error finding container 5c86154db19f4bb5807909548dd01dd6be9da8793248d43df03ae21bf88c57f8: Status 404 returned error can't find the container with id 5c86154db19f4bb5807909548dd01dd6be9da8793248d43df03ae21bf88c57f8
Sep 15 19:01:33.266056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:33.266023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:33.717978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:33.717944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4" event={"ID":"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c","Type":"ContainerStarted","Data":"5c86154db19f4bb5807909548dd01dd6be9da8793248d43df03ae21bf88c57f8"}
Sep 15 19:01:34.586966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.586931    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"]
Sep 15 19:01:34.590370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.590350    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:34.593930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.593911    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-l5kd4\""
Sep 15 19:01:34.594261 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.594241    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 19:01:34.604408 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.604381    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"]
Sep 15 19:01:34.681709 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.681673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpc9\" (UniqueName: \"kubernetes.io/projected/5b8797b1-bc13-486c-bc6b-b65ddd358897-kube-api-access-hjpc9\") pod \"dns-operator-controller-manager-7b84cd7c9c-vp5hc\" (UID: \"5b8797b1-bc13-486c-bc6b-b65ddd358897\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:34.782373 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.782339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpc9\" (UniqueName: \"kubernetes.io/projected/5b8797b1-bc13-486c-bc6b-b65ddd358897-kube-api-access-hjpc9\") pod \"dns-operator-controller-manager-7b84cd7c9c-vp5hc\" (UID: \"5b8797b1-bc13-486c-bc6b-b65ddd358897\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:34.791382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.791349    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpc9\" (UniqueName: \"kubernetes.io/projected/5b8797b1-bc13-486c-bc6b-b65ddd358897-kube-api-access-hjpc9\") pod \"dns-operator-controller-manager-7b84cd7c9c-vp5hc\" (UID: \"5b8797b1-bc13-486c-bc6b-b65ddd358897\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:34.903055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:34.902970    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:36.044000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:36.043969    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"]
Sep 15 19:01:36.044315 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:36.044085    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8797b1_bc13_486c_bc6b_b65ddd358897.slice/crio-c4c65b31d23ba2ddcb4f08fbe12d745131cf3fe9c573e1e6f2ea1be0d7c62fe7 WatchSource:0}: Error finding container c4c65b31d23ba2ddcb4f08fbe12d745131cf3fe9c573e1e6f2ea1be0d7c62fe7: Status 404 returned error can't find the container with id c4c65b31d23ba2ddcb4f08fbe12d745131cf3fe9c573e1e6f2ea1be0d7c62fe7
Sep 15 19:01:36.578474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:36.578440    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:36.738017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:36.737976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc" event={"ID":"5b8797b1-bc13-486c-bc6b-b65ddd358897","Type":"ContainerStarted","Data":"c4c65b31d23ba2ddcb4f08fbe12d745131cf3fe9c573e1e6f2ea1be0d7c62fe7"}
Sep 15 19:01:36.739559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:36.739532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4" event={"ID":"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c","Type":"ContainerStarted","Data":"f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8"}
Sep 15 19:01:36.739674 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:36.739657    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:36.754849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:36.754805    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4" podStartSLOduration=2.006127581 podStartE2EDuration="4.754789886s" totalImagesPullingTime="2.748662305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:36.754317599 +0000 UTC m=+648.908686068" watchObservedRunningTime="2026-09-15 19:01:36.754789886 +0000 UTC m=+648.909158355"
Sep 15 19:01:37.300679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.300641    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4c8xk"]
Sep 15 19:01:37.304302 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.304283    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:37.306571 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.306542    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jwnpt\""
Sep 15 19:01:37.317206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.317179    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4c8xk"]
Sep 15 19:01:37.407382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.407351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnlkp\" (UniqueName: \"kubernetes.io/projected/67c3cca0-8c9f-440f-8367-b8d10e9f3325-kube-api-access-fnlkp\") pod \"authorino-operator-76b478bd78-4c8xk\" (UID: \"67c3cca0-8c9f-440f-8367-b8d10e9f3325\") " pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:37.508206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.508168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnlkp\" (UniqueName: \"kubernetes.io/projected/67c3cca0-8c9f-440f-8367-b8d10e9f3325-kube-api-access-fnlkp\") pod \"authorino-operator-76b478bd78-4c8xk\" (UID: \"67c3cca0-8c9f-440f-8367-b8d10e9f3325\") " pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:37.515738 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.515711    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnlkp\" (UniqueName: \"kubernetes.io/projected/67c3cca0-8c9f-440f-8367-b8d10e9f3325-kube-api-access-fnlkp\") pod \"authorino-operator-76b478bd78-4c8xk\" (UID: \"67c3cca0-8c9f-440f-8367-b8d10e9f3325\") " pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:37.615295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.615209    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:37.897112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:37.897086    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-4c8xk"]
Sep 15 19:01:37.899107 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:37.899063    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c3cca0_8c9f_440f_8367_b8d10e9f3325.slice/crio-b2448b2288b0c4f69940c8be3dd804e4586084a874665f64762f42cf499b7666 WatchSource:0}: Error finding container b2448b2288b0c4f69940c8be3dd804e4586084a874665f64762f42cf499b7666: Status 404 returned error can't find the container with id b2448b2288b0c4f69940c8be3dd804e4586084a874665f64762f42cf499b7666
Sep 15 19:01:38.384158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.384126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:38.485331 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.485298    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:38.643427 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.643313    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:38.755629 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.755581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:38.760185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.760151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk" event={"ID":"67c3cca0-8c9f-440f-8367-b8d10e9f3325","Type":"ContainerStarted","Data":"b2448b2288b0c4f69940c8be3dd804e4586084a874665f64762f42cf499b7666"}
Sep 15 19:01:38.762359 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.762324    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc" event={"ID":"5b8797b1-bc13-486c-bc6b-b65ddd358897","Type":"ContainerStarted","Data":"32220a38160a68c88fb727fc4f837192b4c12ac416244579dbb86c711c0ae3e8"}
Sep 15 19:01:38.762546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.762460    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:38.783474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:38.783384    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc" podStartSLOduration=2.318632932 podStartE2EDuration="4.783363923s" totalImagesPullingTime="2.464730991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:38.78007001 +0000 UTC m=+650.934438479" watchObservedRunningTime="2026-09-15 19:01:38.783363923 +0000 UTC m=+650.937732393"
Sep 15 19:01:39.488560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:39.488527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:39.769942 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:39.769910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk" event={"ID":"67c3cca0-8c9f-440f-8367-b8d10e9f3325","Type":"ContainerStarted","Data":"cd126bb1411ed40b6634fae2115f74977e3fdf70969661a5fe5ad7e550694ca3"}
Sep 15 19:01:39.770110 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:39.769987    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:39.789759 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:39.789708    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk" podStartSLOduration=1.313781276 podStartE2EDuration="2.78969093s" totalImagesPullingTime="1.475909654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:39.788044317 +0000 UTC m=+651.942412785" watchObservedRunningTime="2026-09-15 19:01:39.78969093 +0000 UTC m=+651.944059397"
Sep 15 19:01:41.522154 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.522118    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"]
Sep 15 19:01:41.525549 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.525533    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.529508 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.529488    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tk6j7\""
Sep 15 19:01:41.537223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.537200    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"]
Sep 15 19:01:41.649303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.649259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v892x\" (UniqueName: \"kubernetes.io/projected/d761c32f-c9e0-4cce-a773-5887a78b35b0-kube-api-access-v892x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.649462 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.649321    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d761c32f-c9e0-4cce-a773-5887a78b35b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.749903 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.749868    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v892x\" (UniqueName: \"kubernetes.io/projected/d761c32f-c9e0-4cce-a773-5887a78b35b0-kube-api-access-v892x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.750051 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.749918    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d761c32f-c9e0-4cce-a773-5887a78b35b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.750284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.750266    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d761c32f-c9e0-4cce-a773-5887a78b35b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.757520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.757493    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v892x\" (UniqueName: \"kubernetes.io/projected/d761c32f-c9e0-4cce-a773-5887a78b35b0-kube-api-access-v892x\") pod \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:41.835336 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:41.835270    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:42.110968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:42.110943    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"]
Sep 15 19:01:42.786888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:42.786854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj" event={"ID":"d761c32f-c9e0-4cce-a773-5887a78b35b0","Type":"ContainerStarted","Data":"c907921140c377e28bc2ab9add5e11aec230aebec809bae667abaf878fc79275"}
Sep 15 19:01:42.847583 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:42.847550    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:42.956345 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:42.956314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:45.849421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:45.849385    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:46.647766 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:46.647731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:46.812887 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:46.812855    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj" event={"ID":"d761c32f-c9e0-4cce-a773-5887a78b35b0","Type":"ContainerStarted","Data":"e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980"}
Sep 15 19:01:46.813072 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:46.812975    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:46.834278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:46.834225    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj" podStartSLOduration=1.812759728 podStartE2EDuration="5.834205983s" totalImagesPullingTime="4.021446255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:46.833313628 +0000 UTC m=+658.987682135" watchObservedRunningTime="2026-09-15 19:01:46.834205983 +0000 UTC m=+658.988574451"
Sep 15 19:01:47.748413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:47.748383    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:01:48.885797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:48.885764    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"]
Sep 15 19:01:48.889588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:48.889561    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:48.892349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:48.892040    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 19:01:48.892349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:48.892091    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jsd5d\""
Sep 15 19:01:48.892349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:48.892040    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 19:01:48.894677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:48.894654    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"]
Sep 15 19:01:49.014033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.013997    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.014211 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.014079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qsdm\" (UniqueName: \"kubernetes.io/projected/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-kube-api-access-4qsdm\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.014211 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.014115    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.115252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.115220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsdm\" (UniqueName: \"kubernetes.io/projected/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-kube-api-access-4qsdm\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.115440 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.115373    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.115440 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.115433    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.115566 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:49.115506    2584 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 19:01:49.115641 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:49.115592    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-plugin-serving-cert podName:8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80 nodeName:}" failed. No retries permitted until 2026-09-15 19:01:49.61557583 +0000 UTC m=+661.769944277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-7nplm" (UID: "8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80") : secret "plugin-serving-cert" not found
Sep 15 19:01:49.116273 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.116252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.124172 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.124145    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsdm\" (UniqueName: \"kubernetes.io/projected/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-kube-api-access-4qsdm\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.507286 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.507254    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:49.561309 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.561272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:49.620395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.620363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.622749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.622726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7nplm\" (UID: \"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:49.772241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.772166    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vp5hc"
Sep 15 19:01:49.800217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:49.800186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"
Sep 15 19:01:50.072204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:50.072182    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm"]
Sep 15 19:01:50.074267 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:50.074241    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dd9c656_57b0_4a2c_a51e_f7e6fbbcdf80.slice/crio-966561237ace1374ff2f641909ed0315dd578c68bc8f6b75e179f15cea65fc1f WatchSource:0}: Error finding container 966561237ace1374ff2f641909ed0315dd578c68bc8f6b75e179f15cea65fc1f: Status 404 returned error can't find the container with id 966561237ace1374ff2f641909ed0315dd578c68bc8f6b75e179f15cea65fc1f
Sep 15 19:01:50.777685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:50.777650    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-4c8xk"
Sep 15 19:01:50.841595 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:50.841552    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm" event={"ID":"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80","Type":"ContainerStarted","Data":"966561237ace1374ff2f641909ed0315dd578c68bc8f6b75e179f15cea65fc1f"}
Sep 15 19:01:50.860822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:50.860791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:50.972485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:50.972455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:54.868961 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:54.868923    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm" event={"ID":"8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80","Type":"ContainerStarted","Data":"71b410b7cc9ae7b3b46df817e4486e73db5f0b29520f456aec21428b94d08ede"}
Sep 15 19:01:54.885296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:54.885246    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7nplm" podStartSLOduration=2.296994888 podStartE2EDuration="6.885229249s" totalImagesPullingTime="4.588234361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:01:54.883471754 +0000 UTC m=+667.037840222" watchObservedRunningTime="2026-09-15 19:01:54.885229249 +0000 UTC m=+667.039597714"
Sep 15 19:01:55.591871 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:55.591835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:55.688886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:55.688852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:55.789115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:55.789086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:01:57.824233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:57.824199    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:58.991845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:58.991811    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"]
Sep 15 19:01:58.996556 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:58.996535    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.009560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.009529    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"]
Sep 15 19:01:59.097901 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.097864    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"]
Sep 15 19:01:59.098165 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.098142    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-48rdk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb"
Sep 15 19:01:59.100589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.100560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rdk\" (UniqueName: \"kubernetes.io/projected/bde1abc4-f6aa-463b-8201-4c34842f36cb-kube-api-access-48rdk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.100721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.100605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bde1abc4-f6aa-463b-8201-4c34842f36cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.140236 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.140208    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"]
Sep 15 19:01:59.143585 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.143568    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.157911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.157890    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"]
Sep 15 19:01:59.201743 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.201716    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bde1abc4-f6aa-463b-8201-4c34842f36cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.201845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.201750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j264j\" (UniqueName: \"kubernetes.io/projected/4ffd4f49-ea5c-48fa-9f23-083b92977634-kube-api-access-j264j\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.201845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.201830    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ffd4f49-ea5c-48fa-9f23-083b92977634-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.201932 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.201866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rdk\" (UniqueName: \"kubernetes.io/projected/bde1abc4-f6aa-463b-8201-4c34842f36cb-kube-api-access-48rdk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.202040 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.202020    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bde1abc4-f6aa-463b-8201-4c34842f36cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.209888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.209858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rdk\" (UniqueName: \"kubernetes.io/projected/bde1abc4-f6aa-463b-8201-4c34842f36cb-kube-api-access-48rdk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.302999 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.302975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ffd4f49-ea5c-48fa-9f23-083b92977634-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.303146 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.303036    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j264j\" (UniqueName: \"kubernetes.io/projected/4ffd4f49-ea5c-48fa-9f23-083b92977634-kube-api-access-j264j\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.303371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.303348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ffd4f49-ea5c-48fa-9f23-083b92977634-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.313867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.313841    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j264j\" (UniqueName: \"kubernetes.io/projected/4ffd4f49-ea5c-48fa-9f23-083b92977634-kube-api-access-j264j\") pod \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.454030 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.453983    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.728524 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.728499    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"]
Sep 15 19:01:59.729735 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:01:59.729710    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffd4f49_ea5c_48fa_9f23_083b92977634.slice/crio-17e255bd224363225f08918ff68951f4700f9aa4d725345d5688ddc4f4a01073 WatchSource:0}: Error finding container 17e255bd224363225f08918ff68951f4700f9aa4d725345d5688ddc4f4a01073: Status 404 returned error can't find the container with id 17e255bd224363225f08918ff68951f4700f9aa4d725345d5688ddc4f4a01073
Sep 15 19:01:59.898429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.898392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg" event={"ID":"4ffd4f49-ea5c-48fa-9f23-083b92977634","Type":"ContainerStarted","Data":"06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3"}
Sep 15 19:01:59.898605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.898443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg" event={"ID":"4ffd4f49-ea5c-48fa-9f23-083b92977634","Type":"ContainerStarted","Data":"17e255bd224363225f08918ff68951f4700f9aa4d725345d5688ddc4f4a01073"}
Sep 15 19:01:59.898605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.898479    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.898605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.898479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.905042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.905023    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.910410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.910387    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"]
Sep 15 19:01:59.914641 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.914606    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"]
Sep 15 19:01:59.918072 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.918036    2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4ffd4f49-ea5c-48fa-9f23-083b92977634\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3\\\",\\\"image\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e\\\",\\\"imageID\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:22341c0a543abdcd6cecb58e0e84ffeb713799db22bdf3c01e38cf5c248aee0e\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"300Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-15T19:01:59Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000680000],\\\"uid\\\":1000680000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/kuadrant\\\",\\\"name\\\":\\\"extensions-socket-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-j264j\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.133.0.44\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.133.0.44\\\"}]}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-5f685c496f-rzcdg\": pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.928800 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.928702    2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"bde1abc4-f6aa-463b-8201-4c34842f36cb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-67c5dcb48c-k846h\": pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.928949 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.928889    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"]
Sep 15 19:01:59.931339 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.931319    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"]
Sep 15 19:01:59.931833 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.931811    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.936805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.936782    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"]
Sep 15 19:01:59.937224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.937200    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" containerName="manager"
Sep 15 19:01:59.937224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.937219    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" containerName="manager"
Sep 15 19:01:59.937379 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.937365    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" containerName="manager"
Sep 15 19:01:59.940926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.940908    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"]
Sep 15 19:01:59.941127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.941102    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:01:59.941127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.941104    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" containerName="manager" containerID="cri-o://e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980" gracePeriod=2
Sep 15 19:01:59.955487 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.955391    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"]
Sep 15 19:01:59.959213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.959187    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"]
Sep 15 19:01:59.976110 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.975856    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"]
Sep 15 19:01:59.976643 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.976269    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" containerName="manager" containerID="cri-o://f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8" gracePeriod=2
Sep 15 19:01:59.977022 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.976937    2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d761c32f-c9e0-4cce-a773-5887a78b35b0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\": pods \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:59.979016 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.978986    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:01:59.980969 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.980942    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"]
Sep 15 19:01:59.981336 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.981303    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.981469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.981451    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" containerName="manager"
Sep 15 19:01:59.981534 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.981472    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" containerName="manager"
Sep 15 19:01:59.982196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.981585    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" containerName="manager"
Sep 15 19:01:59.983398 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.983349    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:01:59.985019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.984997    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"]
Sep 15 19:01:59.985159 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.985113    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:01:59.985449 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:01:59.985418    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:01:59.989785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.989720    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"]
Sep 15 19:01:59.993433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.993401    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"]
Sep 15 19:01:59.993924 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.993907    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" containerName="manager"
Sep 15 19:01:59.993992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.993926    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" containerName="manager"
Sep 15 19:01:59.994049 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.994009    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" containerName="manager"
Sep 15 19:01:59.999145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:01:59.999126    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:00.005341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.005315    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"]
Sep 15 19:02:00.008195 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.008165    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mg9b4\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:02:00.008393 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008376    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48rdk\" (UniqueName: \"kubernetes.io/projected/bde1abc4-f6aa-463b-8201-4c34842f36cb-kube-api-access-48rdk\") pod \"bde1abc4-f6aa-463b-8201-4c34842f36cb\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") "
Sep 15 19:02:00.008465 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008450    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bde1abc4-f6aa-463b-8201-4c34842f36cb-extensions-socket-volume\") pod \"bde1abc4-f6aa-463b-8201-4c34842f36cb\" (UID: \"bde1abc4-f6aa-463b-8201-4c34842f36cb\") "
Sep 15 19:02:00.008677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/110e79be-a450-4a30-a1ef-888d88c8e71a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r8k4n\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.008780 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008751    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bde1abc4-f6aa-463b-8201-4c34842f36cb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bde1abc4-f6aa-463b-8201-4c34842f36cb" (UID: "bde1abc4-f6aa-463b-8201-4c34842f36cb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:02:00.008853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a028965-75ef-4c65-baa6-a1cb2988f17c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4x4dk\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.008853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6sk\" (UniqueName: \"kubernetes.io/projected/110e79be-a450-4a30-a1ef-888d88c8e71a-kube-api-access-pg6sk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r8k4n\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.008948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r64c\" (UniqueName: \"kubernetes.io/projected/6a028965-75ef-4c65-baa6-a1cb2988f17c-kube-api-access-7r64c\") pod \"kuadrant-operator-controller-manager-5f685c496f-4x4dk\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.008948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.008937    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bde1abc4-f6aa-463b-8201-4c34842f36cb-extensions-socket-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:00.010393 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.010193    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:00.010685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.010663    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde1abc4-f6aa-463b-8201-4c34842f36cb-kube-api-access-48rdk" (OuterVolumeSpecName: "kube-api-access-48rdk") pod "bde1abc4-f6aa-463b-8201-4c34842f36cb" (UID: "bde1abc4-f6aa-463b-8201-4c34842f36cb"). InnerVolumeSpecName "kube-api-access-48rdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:00.012071 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.012049    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:02:00.013899 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.013879    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mg9b4\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:02:00.016437 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.016418    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:00.050959 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.050920    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:00.109509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.109475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r64c\" (UniqueName: \"kubernetes.io/projected/6a028965-75ef-4c65-baa6-a1cb2988f17c-kube-api-access-7r64c\") pod \"kuadrant-operator-controller-manager-5f685c496f-4x4dk\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.109662 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.109551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fgdj\" (UniqueName: \"kubernetes.io/projected/a479125c-49b3-4e7c-9ab0-a679ae21906d-kube-api-access-7fgdj\") pod \"limitador-operator-controller-manager-74495c69f8-r4dzv\" (UID: \"a479125c-49b3-4e7c-9ab0-a679ae21906d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:00.109662 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.109589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/110e79be-a450-4a30-a1ef-888d88c8e71a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r8k4n\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.109755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.109689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a028965-75ef-4c65-baa6-a1cb2988f17c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4x4dk\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.109755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.109730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6sk\" (UniqueName: \"kubernetes.io/projected/110e79be-a450-4a30-a1ef-888d88c8e71a-kube-api-access-pg6sk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r8k4n\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.109829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.109779    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rdk\" (UniqueName: \"kubernetes.io/projected/bde1abc4-f6aa-463b-8201-4c34842f36cb-kube-api-access-48rdk\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:00.110139 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.110107    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/110e79be-a450-4a30-a1ef-888d88c8e71a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r8k4n\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.110211 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.110180    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a028965-75ef-4c65-baa6-a1cb2988f17c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4x4dk\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.118466 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.118438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6sk\" (UniqueName: \"kubernetes.io/projected/110e79be-a450-4a30-a1ef-888d88c8e71a-kube-api-access-pg6sk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-r8k4n\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.118796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.118777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r64c\" (UniqueName: \"kubernetes.io/projected/6a028965-75ef-4c65-baa6-a1cb2988f17c-kube-api-access-7r64c\") pod \"kuadrant-operator-controller-manager-5f685c496f-4x4dk\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.207607 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.207581    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:02:00.210234 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.210206    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:02:00.210438 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.210415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fgdj\" (UniqueName: \"kubernetes.io/projected/a479125c-49b3-4e7c-9ab0-a679ae21906d-kube-api-access-7fgdj\") pod \"limitador-operator-controller-manager-74495c69f8-r4dzv\" (UID: \"a479125c-49b3-4e7c-9ab0-a679ae21906d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:00.212853 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.212832    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mg9b4\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:02:00.214788 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.214765    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:00.217086 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.217057    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:00.227262 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.227205    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fgdj\" (UniqueName: \"kubernetes.io/projected/a479125c-49b3-4e7c-9ab0-a679ae21906d-kube-api-access-7fgdj\") pod \"limitador-operator-controller-manager-74495c69f8-r4dzv\" (UID: \"a479125c-49b3-4e7c-9ab0-a679ae21906d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:00.230926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.230908    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:02:00.233314 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.233290    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" path="/var/lib/kubelet/pods/bde1abc4-f6aa-463b-8201-4c34842f36cb/volumes"
Sep 15 19:02:00.233486 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.233446    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mg9b4\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:02:00.235572 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.235549    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:00.237598 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.237576    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:00.240232 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.240195    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-g9lwj\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:02:00.255831 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.255801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.311112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.311082    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzt96\" (UniqueName: \"kubernetes.io/projected/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c-kube-api-access-xzt96\") pod \"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c\" (UID: \"1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c\") "
Sep 15 19:02:00.311261 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.311180    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v892x\" (UniqueName: \"kubernetes.io/projected/d761c32f-c9e0-4cce-a773-5887a78b35b0-kube-api-access-v892x\") pod \"d761c32f-c9e0-4cce-a773-5887a78b35b0\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") "
Sep 15 19:02:00.311327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.311280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d761c32f-c9e0-4cce-a773-5887a78b35b0-extensions-socket-volume\") pod \"d761c32f-c9e0-4cce-a773-5887a78b35b0\" (UID: \"d761c32f-c9e0-4cce-a773-5887a78b35b0\") "
Sep 15 19:02:00.311827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.311801    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d761c32f-c9e0-4cce-a773-5887a78b35b0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d761c32f-c9e0-4cce-a773-5887a78b35b0" (UID: "d761c32f-c9e0-4cce-a773-5887a78b35b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:02:00.316909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.316859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d761c32f-c9e0-4cce-a773-5887a78b35b0-kube-api-access-v892x" (OuterVolumeSpecName: "kube-api-access-v892x") pod "d761c32f-c9e0-4cce-a773-5887a78b35b0" (UID: "d761c32f-c9e0-4cce-a773-5887a78b35b0"). InnerVolumeSpecName "kube-api-access-v892x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:00.318555 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.318510    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c-kube-api-access-xzt96" (OuterVolumeSpecName: "kube-api-access-xzt96") pod "1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" (UID: "1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c"). InnerVolumeSpecName "kube-api-access-xzt96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:00.353206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.353174    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.359412 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.359388    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:00.413149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.413055    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v892x\" (UniqueName: \"kubernetes.io/projected/d761c32f-c9e0-4cce-a773-5887a78b35b0-kube-api-access-v892x\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:00.413149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.413091    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d761c32f-c9e0-4cce-a773-5887a78b35b0-extensions-socket-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:00.413149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.413108    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzt96\" (UniqueName: \"kubernetes.io/projected/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c-kube-api-access-xzt96\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:00.530323 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.530297    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"]
Sep 15 19:02:00.532140 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:02:00.532113    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a028965_75ef_4c65_baa6_a1cb2988f17c.slice/crio-62f9ef203ee35343b2c9e1c03d6f14fafaf21c4322661230e5ee2069804e79da WatchSource:0}: Error finding container 62f9ef203ee35343b2c9e1c03d6f14fafaf21c4322661230e5ee2069804e79da: Status 404 returned error can't find the container with id 62f9ef203ee35343b2c9e1c03d6f14fafaf21c4322661230e5ee2069804e79da
Sep 15 19:02:00.637516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.637485    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"]
Sep 15 19:02:00.639529 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:02:00.639500    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110e79be_a450_4a30_a1ef_888d88c8e71a.slice/crio-1328f65237d09e96823bd4f621b579459ebcf72f3188ab2234b4f79f14b2b055 WatchSource:0}: Error finding container 1328f65237d09e96823bd4f621b579459ebcf72f3188ab2234b4f79f14b2b055: Status 404 returned error can't find the container with id 1328f65237d09e96823bd4f621b579459ebcf72f3188ab2234b4f79f14b2b055
Sep 15 19:02:00.640874 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.640794    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"]
Sep 15 19:02:00.642070 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:02:00.642043    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda479125c_49b3_4e7c_9ab0_a679ae21906d.slice/crio-b98fea2c614fefdfdc32e49dac3923216ec5d4fa5591b8d1b711efd850a69554 WatchSource:0}: Error finding container b98fea2c614fefdfdc32e49dac3923216ec5d4fa5591b8d1b711efd850a69554: Status 404 returned error can't find the container with id b98fea2c614fefdfdc32e49dac3923216ec5d4fa5591b8d1b711efd850a69554
Sep 15 19:02:00.905509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.905418    2584 generic.go:356] "Generic (PLEG): container finished" podID="d761c32f-c9e0-4cce-a773-5887a78b35b0" containerID="e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980" exitCode=0
Sep 15 19:02:00.905509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.905466    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-g9lwj"
Sep 15 19:02:00.905777 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.905516    2584 scope.go:117] "RemoveContainer" containerID="e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980"
Sep 15 19:02:00.906864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.906841    2584 generic.go:356] "Generic (PLEG): container finished" podID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" containerID="f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8" exitCode=0
Sep 15 19:02:00.906939 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.906881    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mg9b4"
Sep 15 19:02:00.909041 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.908893    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk" event={"ID":"6a028965-75ef-4c65-baa6-a1cb2988f17c","Type":"ContainerStarted","Data":"e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904"}
Sep 15 19:02:00.909041 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.908956    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk" event={"ID":"6a028965-75ef-4c65-baa6-a1cb2988f17c","Type":"ContainerStarted","Data":"62f9ef203ee35343b2c9e1c03d6f14fafaf21c4322661230e5ee2069804e79da"}
Sep 15 19:02:00.909041 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.908988    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:00.910875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.910849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv" event={"ID":"a479125c-49b3-4e7c-9ab0-a679ae21906d","Type":"ContainerStarted","Data":"6da864fb43b960f3985cf5b55a7d3172dc5909c6699ec395eac6e36e0d0ec53f"}
Sep 15 19:02:00.910981 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.910903    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:00.910981 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.910927    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv" event={"ID":"a479125c-49b3-4e7c-9ab0-a679ae21906d","Type":"ContainerStarted","Data":"b98fea2c614fefdfdc32e49dac3923216ec5d4fa5591b8d1b711efd850a69554"}
Sep 15 19:02:00.913051 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.913029    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n" event={"ID":"110e79be-a450-4a30-a1ef-888d88c8e71a","Type":"ContainerStarted","Data":"fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989"}
Sep 15 19:02:00.913153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.913068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n" event={"ID":"110e79be-a450-4a30-a1ef-888d88c8e71a","Type":"ContainerStarted","Data":"1328f65237d09e96823bd4f621b579459ebcf72f3188ab2234b4f79f14b2b055"}
Sep 15 19:02:00.913153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.913086    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:00.913153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.913113    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:00.913328 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.913243    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg" container="manager"
Sep 15 19:02:00.916940 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.916917    2584 scope.go:117] "RemoveContainer" containerID="e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980"
Sep 15 19:02:00.917222 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.917200    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980\": container with ID starting with e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980 not found: ID does not exist" containerID="e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980"
Sep 15 19:02:00.917309 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.917230    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980"} err="failed to get container status \"e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980\": rpc error: code = NotFound desc = could not find container \"e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980\": container with ID starting with e675f0b980b6dff48fb2e9b8824abcfd7c89954333d220859ae097cb8c0dc980 not found: ID does not exist"
Sep 15 19:02:00.917309 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.917268    2584 scope.go:117] "RemoveContainer" containerID="f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8"
Sep 15 19:02:00.927169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.927119    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk" podStartSLOduration=1.927099275 podStartE2EDuration="1.927099275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:00.926001237 +0000 UTC m=+673.080369707" watchObservedRunningTime="2026-09-15 19:02:00.927099275 +0000 UTC m=+673.081467745"
Sep 15 19:02:00.930312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.930293    2584 scope.go:117] "RemoveContainer" containerID="f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8"
Sep 15 19:02:00.930715 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.930693    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8\": container with ID starting with f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8 not found: ID does not exist" containerID="f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8"
Sep 15 19:02:00.930813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.930720    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8"} err="failed to get container status \"f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8\": rpc error: code = NotFound desc = could not find container \"f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8\": container with ID starting with f7b988c76d24b07380bf4b062e8ba79bbc4660482b08c5fe9b75ec30a8273da8 not found: ID does not exist"
Sep 15 19:02:00.948295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.948250    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv" podStartSLOduration=1.9482326269999999 podStartE2EDuration="1.948232627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:00.94733915 +0000 UTC m=+673.101707618" watchObservedRunningTime="2026-09-15 19:02:00.948232627 +0000 UTC m=+673.102601095"
Sep 15 19:02:00.949357 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:00.949333    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:00.967937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:00.967889    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n" podStartSLOduration=1.9678706049999999 podStartE2EDuration="1.967870605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:00.967012716 +0000 UTC m=+673.121381201" watchObservedRunningTime="2026-09-15 19:02:00.967870605 +0000 UTC m=+673.122239073"
Sep 15 19:02:01.922277 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:01.922239    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" containerName="manager" containerID="cri-o://06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3" gracePeriod=2
Sep 15 19:02:01.924432 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:01.924408    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:01.926206 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:01.926183    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:02.176721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.176658    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:02.179584 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:02.179556    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-k846h\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="bde1abc4-f6aa-463b-8201-4c34842f36cb" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-k846h"
Sep 15 19:02:02.181463 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:02.181444    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:02.233361 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.233333    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c" path="/var/lib/kubelet/pods/1ebd3ffa-6ac2-4cbd-b825-2b4e89569d0c/volumes"
Sep 15 19:02:02.233755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.233742    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d761c32f-c9e0-4cce-a773-5887a78b35b0" path="/var/lib/kubelet/pods/d761c32f-c9e0-4cce-a773-5887a78b35b0/volumes"
Sep 15 19:02:02.242244 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.242208    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ffd4f49-ea5c-48fa-9f23-083b92977634-extensions-socket-volume\") pod \"4ffd4f49-ea5c-48fa-9f23-083b92977634\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") "
Sep 15 19:02:02.242353 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.242261    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j264j\" (UniqueName: \"kubernetes.io/projected/4ffd4f49-ea5c-48fa-9f23-083b92977634-kube-api-access-j264j\") pod \"4ffd4f49-ea5c-48fa-9f23-083b92977634\" (UID: \"4ffd4f49-ea5c-48fa-9f23-083b92977634\") "
Sep 15 19:02:02.242475 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.242425    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ffd4f49-ea5c-48fa-9f23-083b92977634-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4ffd4f49-ea5c-48fa-9f23-083b92977634" (UID: "4ffd4f49-ea5c-48fa-9f23-083b92977634"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:02:02.242927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.242904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:02.244353 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.244322    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffd4f49-ea5c-48fa-9f23-083b92977634-kube-api-access-j264j" (OuterVolumeSpecName: "kube-api-access-j264j") pod "4ffd4f49-ea5c-48fa-9f23-083b92977634" (UID: "4ffd4f49-ea5c-48fa-9f23-083b92977634"). InnerVolumeSpecName "kube-api-access-j264j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:02.343115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.343092    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j264j\" (UniqueName: \"kubernetes.io/projected/4ffd4f49-ea5c-48fa-9f23-083b92977634-kube-api-access-j264j\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:02.343115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.343113    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ffd4f49-ea5c-48fa-9f23-083b92977634-extensions-socket-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:02.343822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.343802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:02.929338 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.929253    2584 generic.go:356] "Generic (PLEG): container finished" podID="4ffd4f49-ea5c-48fa-9f23-083b92977634" containerID="06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3" exitCode=2
Sep 15 19:02:02.929338 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.929297    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:02.929799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.929359    2584 scope.go:117] "RemoveContainer" containerID="06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3"
Sep 15 19:02:02.939089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.939068    2584 scope.go:117] "RemoveContainer" containerID="06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3"
Sep 15 19:02:02.939368 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:02.939350    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3\": container with ID starting with 06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3 not found: ID does not exist" containerID="06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3"
Sep 15 19:02:02.939430 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:02.939376    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3"} err="failed to get container status \"06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3\": rpc error: code = NotFound desc = could not find container \"06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3\": container with ID starting with 06fe49f5c5d5a10883049dd4900b9743047a4229f99dd555478be7eefb67d2a3 not found: ID does not exist"
Sep 15 19:02:02.940050 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:02.940030    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-rzcdg\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-rzcdg"
Sep 15 19:02:04.233841 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:04.233807    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffd4f49-ea5c-48fa-9f23-083b92977634" path="/var/lib/kubelet/pods/4ffd4f49-ea5c-48fa-9f23-083b92977634/volumes"
Sep 15 19:02:05.235707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:05.235669    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:05.336099 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:05.336066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:11.030606 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.030568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:11.084262 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.084234    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:11.860127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.860093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:11.914699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.914673    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:11.924381 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.924360    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:11.924496 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.924427    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:11.924565 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:11.924556    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r4dzv"
Sep 15 19:02:12.007064 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.007030    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"]
Sep 15 19:02:12.007394 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.007246    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n" podUID="110e79be-a450-4a30-a1ef-888d88c8e71a" containerName="manager" containerID="cri-o://fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989" gracePeriod=10
Sep 15 19:02:12.273246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.273218    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:12.326752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.326722    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg6sk\" (UniqueName: \"kubernetes.io/projected/110e79be-a450-4a30-a1ef-888d88c8e71a-kube-api-access-pg6sk\") pod \"110e79be-a450-4a30-a1ef-888d88c8e71a\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") "
Sep 15 19:02:12.326922 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.326760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/110e79be-a450-4a30-a1ef-888d88c8e71a-extensions-socket-volume\") pod \"110e79be-a450-4a30-a1ef-888d88c8e71a\" (UID: \"110e79be-a450-4a30-a1ef-888d88c8e71a\") "
Sep 15 19:02:12.327124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.327100    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110e79be-a450-4a30-a1ef-888d88c8e71a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "110e79be-a450-4a30-a1ef-888d88c8e71a" (UID: "110e79be-a450-4a30-a1ef-888d88c8e71a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:02:12.328737 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.328704    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110e79be-a450-4a30-a1ef-888d88c8e71a-kube-api-access-pg6sk" (OuterVolumeSpecName: "kube-api-access-pg6sk") pod "110e79be-a450-4a30-a1ef-888d88c8e71a" (UID: "110e79be-a450-4a30-a1ef-888d88c8e71a"). InnerVolumeSpecName "kube-api-access-pg6sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:12.379830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.379802    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"]
Sep 15 19:02:12.380177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.380165    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110e79be-a450-4a30-a1ef-888d88c8e71a" containerName="manager"
Sep 15 19:02:12.380221 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.380178    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="110e79be-a450-4a30-a1ef-888d88c8e71a" containerName="manager"
Sep 15 19:02:12.380263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.380253    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="110e79be-a450-4a30-a1ef-888d88c8e71a" containerName="manager"
Sep 15 19:02:12.396879 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.396812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.398704 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.398676    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"]
Sep 15 19:02:12.427792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.427763    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6sk\" (UniqueName: \"kubernetes.io/projected/110e79be-a450-4a30-a1ef-888d88c8e71a-kube-api-access-pg6sk\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:12.427792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.427786    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/110e79be-a450-4a30-a1ef-888d88c8e71a-extensions-socket-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:12.528888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.528851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvbb\" (UniqueName: \"kubernetes.io/projected/938cd1dd-3ac6-4af7-86e7-eea329eda610-kube-api-access-wjvbb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5jswj\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.529075 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.528951    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938cd1dd-3ac6-4af7-86e7-eea329eda610-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5jswj\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.629410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.629370    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938cd1dd-3ac6-4af7-86e7-eea329eda610-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5jswj\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.629586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.629462    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvbb\" (UniqueName: \"kubernetes.io/projected/938cd1dd-3ac6-4af7-86e7-eea329eda610-kube-api-access-wjvbb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5jswj\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.629791 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.629769    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938cd1dd-3ac6-4af7-86e7-eea329eda610-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5jswj\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.643520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.643498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvbb\" (UniqueName: \"kubernetes.io/projected/938cd1dd-3ac6-4af7-86e7-eea329eda610-kube-api-access-wjvbb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5jswj\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.708442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.708357    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:12.990248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.990169    2584 generic.go:356] "Generic (PLEG): container finished" podID="110e79be-a450-4a30-a1ef-888d88c8e71a" containerID="fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989" exitCode=0
Sep 15 19:02:12.990248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.990211    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"
Sep 15 19:02:12.990248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.990221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n" event={"ID":"110e79be-a450-4a30-a1ef-888d88c8e71a","Type":"ContainerDied","Data":"fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989"}
Sep 15 19:02:12.990520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.990261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n" event={"ID":"110e79be-a450-4a30-a1ef-888d88c8e71a","Type":"ContainerDied","Data":"1328f65237d09e96823bd4f621b579459ebcf72f3188ab2234b4f79f14b2b055"}
Sep 15 19:02:12.990520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.990284    2584 scope.go:117] "RemoveContainer" containerID="fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989"
Sep 15 19:02:12.999062 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.999046    2584 scope.go:117] "RemoveContainer" containerID="fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989"
Sep 15 19:02:12.999314 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:12.999298    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989\": container with ID starting with fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989 not found: ID does not exist" containerID="fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989"
Sep 15 19:02:12.999360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:12.999323    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989"} err="failed to get container status \"fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989\": rpc error: code = NotFound desc = could not find container \"fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989\": container with ID starting with fcca0be6b12c095222162159c5d9a14182f21eaf28085ec0e4d23c139322c989 not found: ID does not exist"
Sep 15 19:02:13.016699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:13.016679    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"]
Sep 15 19:02:13.018852 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:02:13.018830    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938cd1dd_3ac6_4af7_86e7_eea329eda610.slice/crio-ccd72ef7f7c58452ff3568ffd066fe27e01267b4b646e65a2630e08b208bc0cf WatchSource:0}: Error finding container ccd72ef7f7c58452ff3568ffd066fe27e01267b4b646e65a2630e08b208bc0cf: Status 404 returned error can't find the container with id ccd72ef7f7c58452ff3568ffd066fe27e01267b4b646e65a2630e08b208bc0cf
Sep 15 19:02:13.033821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:13.033803    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"]
Sep 15 19:02:13.049167 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:13.049148    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-r8k4n"]
Sep 15 19:02:13.998859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:13.998821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj" event={"ID":"938cd1dd-3ac6-4af7-86e7-eea329eda610","Type":"ContainerStarted","Data":"7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a"}
Sep 15 19:02:13.999238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:13.998925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj" event={"ID":"938cd1dd-3ac6-4af7-86e7-eea329eda610","Type":"ContainerStarted","Data":"ccd72ef7f7c58452ff3568ffd066fe27e01267b4b646e65a2630e08b208bc0cf"}
Sep 15 19:02:13.999238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:13.998955    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:14.017385 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:14.017334    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj" podStartSLOduration=2.017316618 podStartE2EDuration="2.017316618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:14.016274483 +0000 UTC m=+686.170642975" watchObservedRunningTime="2026-09-15 19:02:14.017316618 +0000 UTC m=+686.171685088"
Sep 15 19:02:14.233807 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:14.233774    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110e79be-a450-4a30-a1ef-888d88c8e71a" path="/var/lib/kubelet/pods/110e79be-a450-4a30-a1ef-888d88c8e71a/volumes"
Sep 15 19:02:15.100779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:15.100750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:15.197624 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:15.197590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:15.254316 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:15.254286    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:16.828009 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:16.827980    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:16.928525 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:16.928489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:25.006964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.006934    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:02:25.057372 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.057328    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"]
Sep 15 19:02:25.057851 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.057816    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk" podUID="6a028965-75ef-4c65-baa6-a1cb2988f17c" containerName="manager" containerID="cri-o://e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904" gracePeriod=10
Sep 15 19:02:25.312444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.312418    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:25.442228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.442197    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r64c\" (UniqueName: \"kubernetes.io/projected/6a028965-75ef-4c65-baa6-a1cb2988f17c-kube-api-access-7r64c\") pod \"6a028965-75ef-4c65-baa6-a1cb2988f17c\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") "
Sep 15 19:02:25.442376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.442285    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a028965-75ef-4c65-baa6-a1cb2988f17c-extensions-socket-volume\") pod \"6a028965-75ef-4c65-baa6-a1cb2988f17c\" (UID: \"6a028965-75ef-4c65-baa6-a1cb2988f17c\") "
Sep 15 19:02:25.442651 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.442628    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a028965-75ef-4c65-baa6-a1cb2988f17c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a028965-75ef-4c65-baa6-a1cb2988f17c" (UID: "6a028965-75ef-4c65-baa6-a1cb2988f17c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:02:25.444099 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.444075    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a028965-75ef-4c65-baa6-a1cb2988f17c-kube-api-access-7r64c" (OuterVolumeSpecName: "kube-api-access-7r64c") pod "6a028965-75ef-4c65-baa6-a1cb2988f17c" (UID: "6a028965-75ef-4c65-baa6-a1cb2988f17c"). InnerVolumeSpecName "kube-api-access-7r64c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:25.543741 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.543672    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a028965-75ef-4c65-baa6-a1cb2988f17c-extensions-socket-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:25.543741 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:25.543702    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r64c\" (UniqueName: \"kubernetes.io/projected/6a028965-75ef-4c65-baa6-a1cb2988f17c-kube-api-access-7r64c\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:26.071563 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.071534    2584 generic.go:356] "Generic (PLEG): container finished" podID="6a028965-75ef-4c65-baa6-a1cb2988f17c" containerID="e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904" exitCode=0
Sep 15 19:02:26.071962 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.071581    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"
Sep 15 19:02:26.071962 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.071636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk" event={"ID":"6a028965-75ef-4c65-baa6-a1cb2988f17c","Type":"ContainerDied","Data":"e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904"}
Sep 15 19:02:26.071962 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.071690    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk" event={"ID":"6a028965-75ef-4c65-baa6-a1cb2988f17c","Type":"ContainerDied","Data":"62f9ef203ee35343b2c9e1c03d6f14fafaf21c4322661230e5ee2069804e79da"}
Sep 15 19:02:26.071962 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.071721    2584 scope.go:117] "RemoveContainer" containerID="e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904"
Sep 15 19:02:26.081026 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.081010    2584 scope.go:117] "RemoveContainer" containerID="e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904"
Sep 15 19:02:26.081298 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:26.081281    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904\": container with ID starting with e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904 not found: ID does not exist" containerID="e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904"
Sep 15 19:02:26.081338 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.081306    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904"} err="failed to get container status \"e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904\": rpc error: code = NotFound desc = could not find container \"e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904\": container with ID starting with e402aa9feee48ec21632b5d6ef63c48ec911109617fe13b85628279d30651904 not found: ID does not exist"
Sep 15 19:02:26.093153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.093131    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"]
Sep 15 19:02:26.101845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.101821    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4x4dk"]
Sep 15 19:02:26.233572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:26.233546    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a028965-75ef-4c65-baa6-a1cb2988f17c" path="/var/lib/kubelet/pods/6a028965-75ef-4c65-baa6-a1cb2988f17c/volumes"
Sep 15 19:02:28.288025 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.287994    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"]
Sep 15 19:02:28.288486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.288360    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a028965-75ef-4c65-baa6-a1cb2988f17c" containerName="manager"
Sep 15 19:02:28.288486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.288375    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a028965-75ef-4c65-baa6-a1cb2988f17c" containerName="manager"
Sep 15 19:02:28.288486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.288432    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a028965-75ef-4c65-baa6-a1cb2988f17c" containerName="manager"
Sep 15 19:02:28.295906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.295885    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.298338 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.298317    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kwtfr\""
Sep 15 19:02:28.303283 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.303261    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"]
Sep 15 19:02:28.310834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.310804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:28.411131 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.411098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:28.469448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469594    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469642    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469709    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f9c\" (UniqueName: \"kubernetes.io/projected/0eb5d451-4682-4cf9-ac47-60323346ce64-kube-api-access-b6f9c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469739    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb5d451-4682-4cf9-ac47-60323346ce64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.469937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.469815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570598    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f9c\" (UniqueName: \"kubernetes.io/projected/0eb5d451-4682-4cf9-ac47-60323346ce64-kube-api-access-b6f9c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb5d451-4682-4cf9-ac47-60323346ce64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570827    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.570941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.570875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.571253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.571099    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.571253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.571144    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.571253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.571178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.571253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.571243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.571569 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.571553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eb5d451-4682-4cf9-ac47-60323346ce64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.572950 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.572927    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.573248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.573229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.578276 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.578250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eb5d451-4682-4cf9-ac47-60323346ce64-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.578356 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.578314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f9c\" (UniqueName: \"kubernetes.io/projected/0eb5d451-4682-4cf9-ac47-60323346ce64-kube-api-access-b6f9c\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zp9s8\" (UID: \"0eb5d451-4682-4cf9-ac47-60323346ce64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.608260 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.608237    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:28.882290 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.882259    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"]
Sep 15 19:02:28.883581 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:02:28.883554    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb5d451_4682_4cf9_ac47_60323346ce64.slice/crio-46abb129198d10c6649624084a1e732626d2737c5d7d275a1841c9e478ed7411 WatchSource:0}: Error finding container 46abb129198d10c6649624084a1e732626d2737c5d7d275a1841c9e478ed7411: Status 404 returned error can't find the container with id 46abb129198d10c6649624084a1e732626d2737c5d7d275a1841c9e478ed7411
Sep 15 19:02:28.889490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.889454    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:02:28.889568 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.889542    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:02:28.889605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:28.889582    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:02:29.092097 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:29.092057    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" event={"ID":"0eb5d451-4682-4cf9-ac47-60323346ce64","Type":"ContainerStarted","Data":"060b4b52df1987d2853eb9ff5686ef9a1e8339f32134c76fca8a7862386de2d8"}
Sep 15 19:02:29.092253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:29.092126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" event={"ID":"0eb5d451-4682-4cf9-ac47-60323346ce64","Type":"ContainerStarted","Data":"46abb129198d10c6649624084a1e732626d2737c5d7d275a1841c9e478ed7411"}
Sep 15 19:02:29.112987 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:29.112908    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" podStartSLOduration=1.112887447 podStartE2EDuration="1.112887447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:29.11073006 +0000 UTC m=+701.265098528" watchObservedRunningTime="2026-09-15 19:02:29.112887447 +0000 UTC m=+701.267255981"
Sep 15 19:02:29.608666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:29.608600    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:29.613455 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:29.613432    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:30.097498 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:30.097468    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:30.098399 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:30.098380    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:02:42.761403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:42.761331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:42.860387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:42.860353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:43.251885 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.251806    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-226q9"]
Sep 15 19:02:43.255500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.255479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:43.257883 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.257854    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mzzjb\""
Sep 15 19:02:43.259736 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.259700    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-226q9"]
Sep 15 19:02:43.304780 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.304748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mkn\" (UniqueName: \"kubernetes.io/projected/55f1437b-9cca-4901-a3b2-02e4da14e460-kube-api-access-v6mkn\") pod \"authorino-7bcfd8f9cd-226q9\" (UID: \"55f1437b-9cca-4901-a3b2-02e4da14e460\") " pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:43.406275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.406237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mkn\" (UniqueName: \"kubernetes.io/projected/55f1437b-9cca-4901-a3b2-02e4da14e460-kube-api-access-v6mkn\") pod \"authorino-7bcfd8f9cd-226q9\" (UID: \"55f1437b-9cca-4901-a3b2-02e4da14e460\") " pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:43.413385 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.413358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mkn\" (UniqueName: \"kubernetes.io/projected/55f1437b-9cca-4901-a3b2-02e4da14e460-kube-api-access-v6mkn\") pod \"authorino-7bcfd8f9cd-226q9\" (UID: \"55f1437b-9cca-4901-a3b2-02e4da14e460\") " pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:43.568075 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.568037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:43.708063 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.708025    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:43.808276 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.808248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:02:43.836328 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.836301    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-226q9"]
Sep 15 19:02:43.837448 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:02:43.837422    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55f1437b_9cca_4901_a3b2_02e4da14e460.slice/crio-30f4c9dd2ba429f5742890912f4b6c598435edb8aa8ac1985cde7e9d194296b8 WatchSource:0}: Error finding container 30f4c9dd2ba429f5742890912f4b6c598435edb8aa8ac1985cde7e9d194296b8: Status 404 returned error can't find the container with id 30f4c9dd2ba429f5742890912f4b6c598435edb8aa8ac1985cde7e9d194296b8
Sep 15 19:02:43.838869 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:43.838848    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:02:44.175564 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:44.175484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9" event={"ID":"55f1437b-9cca-4901-a3b2-02e4da14e460","Type":"ContainerStarted","Data":"30f4c9dd2ba429f5742890912f4b6c598435edb8aa8ac1985cde7e9d194296b8"}
Sep 15 19:02:46.187737 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:46.187644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9" event={"ID":"55f1437b-9cca-4901-a3b2-02e4da14e460","Type":"ContainerStarted","Data":"7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5"}
Sep 15 19:02:46.201773 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:46.201723    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9" podStartSLOduration=1.116529422 podStartE2EDuration="3.20170826s" totalImagesPullingTime="2.085178838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:02:46.200484623 +0000 UTC m=+718.354853091" watchObservedRunningTime="2026-09-15 19:02:46.20170826 +0000 UTC m=+718.356076727"
Sep 15 19:02:47.668245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:47.668207    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-226q9"]
Sep 15 19:02:48.199553 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:48.199502    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9" podUID="55f1437b-9cca-4901-a3b2-02e4da14e460" containerName="authorino" containerID="cri-o://7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5" gracePeriod=30
Sep 15 19:02:48.445415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:48.445393    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:48.549262 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:48.549227    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6mkn\" (UniqueName: \"kubernetes.io/projected/55f1437b-9cca-4901-a3b2-02e4da14e460-kube-api-access-v6mkn\") pod \"55f1437b-9cca-4901-a3b2-02e4da14e460\" (UID: \"55f1437b-9cca-4901-a3b2-02e4da14e460\") "
Sep 15 19:02:48.551305 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:48.551281    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f1437b-9cca-4901-a3b2-02e4da14e460-kube-api-access-v6mkn" (OuterVolumeSpecName: "kube-api-access-v6mkn") pod "55f1437b-9cca-4901-a3b2-02e4da14e460" (UID: "55f1437b-9cca-4901-a3b2-02e4da14e460"). InnerVolumeSpecName "kube-api-access-v6mkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:02:48.650464 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:48.650423    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mkn\" (UniqueName: \"kubernetes.io/projected/55f1437b-9cca-4901-a3b2-02e4da14e460-kube-api-access-v6mkn\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:02:49.206109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.206075    2584 generic.go:356] "Generic (PLEG): container finished" podID="55f1437b-9cca-4901-a3b2-02e4da14e460" containerID="7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5" exitCode=0
Sep 15 19:02:49.206533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.206117    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9"
Sep 15 19:02:49.206533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.206152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9" event={"ID":"55f1437b-9cca-4901-a3b2-02e4da14e460","Type":"ContainerDied","Data":"7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5"}
Sep 15 19:02:49.206533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.206195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-226q9" event={"ID":"55f1437b-9cca-4901-a3b2-02e4da14e460","Type":"ContainerDied","Data":"30f4c9dd2ba429f5742890912f4b6c598435edb8aa8ac1985cde7e9d194296b8"}
Sep 15 19:02:49.206533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.206215    2584 scope.go:117] "RemoveContainer" containerID="7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5"
Sep 15 19:02:49.214987 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.214969    2584 scope.go:117] "RemoveContainer" containerID="7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5"
Sep 15 19:02:49.215286 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:02:49.215260    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5\": container with ID starting with 7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5 not found: ID does not exist" containerID="7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5"
Sep 15 19:02:49.215336 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.215302    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5"} err="failed to get container status \"7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5\": rpc error: code = NotFound desc = could not find container \"7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5\": container with ID starting with 7cdedf6a2a333ab360936676be20396fe87536de27464f27237153669c059aa5 not found: ID does not exist"
Sep 15 19:02:49.226083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.226062    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-226q9"]
Sep 15 19:02:49.228254 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:49.228234    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-226q9"]
Sep 15 19:02:50.233254 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:02:50.233220    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f1437b-9cca-4901-a3b2-02e4da14e460" path="/var/lib/kubelet/pods/55f1437b-9cca-4901-a3b2-02e4da14e460/volumes"
Sep 15 19:03:01.432759 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:01.432720    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:01.532532 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:01.532501    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:01.886546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:01.886507    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:05.131175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:05.131143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:05.234560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:05.234532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:14.060647 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.060594    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:14.282019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.281984    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-97h6s"]
Sep 15 19:03:14.282333 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.282320    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f1437b-9cca-4901-a3b2-02e4da14e460" containerName="authorino"
Sep 15 19:03:14.282378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.282335    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f1437b-9cca-4901-a3b2-02e4da14e460" containerName="authorino"
Sep 15 19:03:14.282413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.282404    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f1437b-9cca-4901-a3b2-02e4da14e460" containerName="authorino"
Sep 15 19:03:14.285307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.285286    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:14.287587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.287569    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mzzjb\""
Sep 15 19:03:14.293865 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.293833    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-97h6s"]
Sep 15 19:03:14.373323 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.373252    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtjd\" (UniqueName: \"kubernetes.io/projected/ec64721e-1809-4646-ac17-1c840891f3ba-kube-api-access-kqtjd\") pod \"authorino-557f876f69-97h6s\" (UID: \"ec64721e-1809-4646-ac17-1c840891f3ba\") " pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:14.474620 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.474576    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtjd\" (UniqueName: \"kubernetes.io/projected/ec64721e-1809-4646-ac17-1c840891f3ba-kube-api-access-kqtjd\") pod \"authorino-557f876f69-97h6s\" (UID: \"ec64721e-1809-4646-ac17-1c840891f3ba\") " pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:14.483827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.483796    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtjd\" (UniqueName: \"kubernetes.io/projected/ec64721e-1809-4646-ac17-1c840891f3ba-kube-api-access-kqtjd\") pod \"authorino-557f876f69-97h6s\" (UID: \"ec64721e-1809-4646-ac17-1c840891f3ba\") " pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:14.500247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.500216    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-97h6s"]
Sep 15 19:03:14.500490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.500474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:14.526130 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.526105    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-587fc46cff-hxmrl"]
Sep 15 19:03:14.530100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.530081    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:14.538133 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.538106    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-587fc46cff-hxmrl"]
Sep 15 19:03:14.604372 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.604344    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587fc46cff-hxmrl"]
Sep 15 19:03:14.604637 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:03:14.604595    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gjtlp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-587fc46cff-hxmrl" podUID="9b982c2f-7303-4ac6-a4e0-82068d51864b"
Sep 15 19:03:14.632948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.632873    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cb7d44c65-6h2jr"]
Sep 15 19:03:14.636291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.636269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.638632 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.638596    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 19:03:14.644286 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.644266    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb7d44c65-6h2jr"]
Sep 15 19:03:14.676525 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.676496    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjtlp\" (UniqueName: \"kubernetes.io/projected/9b982c2f-7303-4ac6-a4e0-82068d51864b-kube-api-access-gjtlp\") pod \"authorino-587fc46cff-hxmrl\" (UID: \"9b982c2f-7303-4ac6-a4e0-82068d51864b\") " pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:14.776600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.776578    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-97h6s"]
Sep 15 19:03:14.776990 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.776973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5n4\" (UniqueName: \"kubernetes.io/projected/2331c363-cccd-4129-a5e2-72e6d4d16ad8-kube-api-access-pt5n4\") pod \"authorino-7cb7d44c65-6h2jr\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") " pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.777042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.777015    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjtlp\" (UniqueName: \"kubernetes.io/projected/9b982c2f-7303-4ac6-a4e0-82068d51864b-kube-api-access-gjtlp\") pod \"authorino-587fc46cff-hxmrl\" (UID: \"9b982c2f-7303-4ac6-a4e0-82068d51864b\") " pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:14.777092 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.777064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2331c363-cccd-4129-a5e2-72e6d4d16ad8-tls-cert\") pod \"authorino-7cb7d44c65-6h2jr\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") " pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.778658 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:03:14.778627    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec64721e_1809_4646_ac17_1c840891f3ba.slice/crio-4bd0f3b7942a329743a6971a62bd3e46f9ebaa51dc631919166906c5ad6caf5f WatchSource:0}: Error finding container 4bd0f3b7942a329743a6971a62bd3e46f9ebaa51dc631919166906c5ad6caf5f: Status 404 returned error can't find the container with id 4bd0f3b7942a329743a6971a62bd3e46f9ebaa51dc631919166906c5ad6caf5f
Sep 15 19:03:14.784521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.784496    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjtlp\" (UniqueName: \"kubernetes.io/projected/9b982c2f-7303-4ac6-a4e0-82068d51864b-kube-api-access-gjtlp\") pod \"authorino-587fc46cff-hxmrl\" (UID: \"9b982c2f-7303-4ac6-a4e0-82068d51864b\") " pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:14.877929 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.877898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5n4\" (UniqueName: \"kubernetes.io/projected/2331c363-cccd-4129-a5e2-72e6d4d16ad8-kube-api-access-pt5n4\") pod \"authorino-7cb7d44c65-6h2jr\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") " pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.878036 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.877942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2331c363-cccd-4129-a5e2-72e6d4d16ad8-tls-cert\") pod \"authorino-7cb7d44c65-6h2jr\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") " pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.880208 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.880191    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2331c363-cccd-4129-a5e2-72e6d4d16ad8-tls-cert\") pod \"authorino-7cb7d44c65-6h2jr\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") " pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.886536 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.886496    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5n4\" (UniqueName: \"kubernetes.io/projected/2331c363-cccd-4129-a5e2-72e6d4d16ad8-kube-api-access-pt5n4\") pod \"authorino-7cb7d44c65-6h2jr\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") " pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:14.947100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:14.947068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:03:15.216448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.216425    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb7d44c65-6h2jr"]
Sep 15 19:03:15.217707 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:03:15.217671    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2331c363_cccd_4129_a5e2_72e6d4d16ad8.slice/crio-68594e6e14b66e3bcd7f6b6bd91e4bd75c01a6d3b78ed21f1fce304fb14104e0 WatchSource:0}: Error finding container 68594e6e14b66e3bcd7f6b6bd91e4bd75c01a6d3b78ed21f1fce304fb14104e0: Status 404 returned error can't find the container with id 68594e6e14b66e3bcd7f6b6bd91e4bd75c01a6d3b78ed21f1fce304fb14104e0
Sep 15 19:03:15.354522 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.354484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr" event={"ID":"2331c363-cccd-4129-a5e2-72e6d4d16ad8","Type":"ContainerStarted","Data":"68594e6e14b66e3bcd7f6b6bd91e4bd75c01a6d3b78ed21f1fce304fb14104e0"}
Sep 15 19:03:15.356329 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.356276    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-97h6s" event={"ID":"ec64721e-1809-4646-ac17-1c840891f3ba","Type":"ContainerStarted","Data":"c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5"}
Sep 15 19:03:15.356329 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.356309    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-97h6s" podUID="ec64721e-1809-4646-ac17-1c840891f3ba" containerName="authorino" containerID="cri-o://c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5" gracePeriod=30
Sep 15 19:03:15.356554 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.356318    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-97h6s" event={"ID":"ec64721e-1809-4646-ac17-1c840891f3ba","Type":"ContainerStarted","Data":"4bd0f3b7942a329743a6971a62bd3e46f9ebaa51dc631919166906c5ad6caf5f"}
Sep 15 19:03:15.356554 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.356321    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:15.363810 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.363791    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:15.372261 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.372216    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-97h6s" podStartSLOduration=1.010042575 podStartE2EDuration="1.372202695s" totalImagesPullingTime="362.16012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:15.370054063 +0000 UTC m=+747.524422531" watchObservedRunningTime="2026-09-15 19:03:15.372202695 +0000 UTC m=+747.526571162"
Sep 15 19:03:15.461016 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.460936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:15.484442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.484416    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjtlp\" (UniqueName: \"kubernetes.io/projected/9b982c2f-7303-4ac6-a4e0-82068d51864b-kube-api-access-gjtlp\") pod \"9b982c2f-7303-4ac6-a4e0-82068d51864b\" (UID: \"9b982c2f-7303-4ac6-a4e0-82068d51864b\") "
Sep 15 19:03:15.486289 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.486261    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b982c2f-7303-4ac6-a4e0-82068d51864b-kube-api-access-gjtlp" (OuterVolumeSpecName: "kube-api-access-gjtlp") pod "9b982c2f-7303-4ac6-a4e0-82068d51864b" (UID: "9b982c2f-7303-4ac6-a4e0-82068d51864b"). InnerVolumeSpecName "kube-api-access-gjtlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:15.585885 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.585861    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjtlp\" (UniqueName: \"kubernetes.io/projected/9b982c2f-7303-4ac6-a4e0-82068d51864b-kube-api-access-gjtlp\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:03:15.604019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.603994    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:15.787404 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.787383    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqtjd\" (UniqueName: \"kubernetes.io/projected/ec64721e-1809-4646-ac17-1c840891f3ba-kube-api-access-kqtjd\") pod \"ec64721e-1809-4646-ac17-1c840891f3ba\" (UID: \"ec64721e-1809-4646-ac17-1c840891f3ba\") "
Sep 15 19:03:15.789224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.789203    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec64721e-1809-4646-ac17-1c840891f3ba-kube-api-access-kqtjd" (OuterVolumeSpecName: "kube-api-access-kqtjd") pod "ec64721e-1809-4646-ac17-1c840891f3ba" (UID: "ec64721e-1809-4646-ac17-1c840891f3ba"). InnerVolumeSpecName "kube-api-access-kqtjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:03:15.888757 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:15.888724    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqtjd\" (UniqueName: \"kubernetes.io/projected/ec64721e-1809-4646-ac17-1c840891f3ba-kube-api-access-kqtjd\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:03:16.363022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.362987    2584 generic.go:356] "Generic (PLEG): container finished" podID="ec64721e-1809-4646-ac17-1c840891f3ba" containerID="c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5" exitCode=0
Sep 15 19:03:16.363472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.363043    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-97h6s"
Sep 15 19:03:16.363472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.363076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-97h6s" event={"ID":"ec64721e-1809-4646-ac17-1c840891f3ba","Type":"ContainerDied","Data":"c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5"}
Sep 15 19:03:16.363472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.363130    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-97h6s" event={"ID":"ec64721e-1809-4646-ac17-1c840891f3ba","Type":"ContainerDied","Data":"4bd0f3b7942a329743a6971a62bd3e46f9ebaa51dc631919166906c5ad6caf5f"}
Sep 15 19:03:16.363472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.363156    2584 scope.go:117] "RemoveContainer" containerID="c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5"
Sep 15 19:03:16.365093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.365055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr" event={"ID":"2331c363-cccd-4129-a5e2-72e6d4d16ad8","Type":"ContainerStarted","Data":"10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054"}
Sep 15 19:03:16.365319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.365167    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc46cff-hxmrl"
Sep 15 19:03:16.372376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.372355    2584 scope.go:117] "RemoveContainer" containerID="c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5"
Sep 15 19:03:16.372649 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:03:16.372627    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5\": container with ID starting with c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5 not found: ID does not exist" containerID="c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5"
Sep 15 19:03:16.372718 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.372658    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5"} err="failed to get container status \"c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5\": rpc error: code = NotFound desc = could not find container \"c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5\": container with ID starting with c9cf9a9355c4527bc77244c6f7897458567b7e253308699d53c9950afcc93bf5 not found: ID does not exist"
Sep 15 19:03:16.378280 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.378257    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-97h6s"]
Sep 15 19:03:16.381776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.381756    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-97h6s"]
Sep 15 19:03:16.396921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.396882    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr" podStartSLOduration=1.868907247 podStartE2EDuration="2.396869102s" totalImagesPullingTime="527.961855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:03:16.394816979 +0000 UTC m=+748.549185449" watchObservedRunningTime="2026-09-15 19:03:16.396869102 +0000 UTC m=+748.551237612"
Sep 15 19:03:16.417171 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.417145    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587fc46cff-hxmrl"]
Sep 15 19:03:16.426263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:16.426238    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-587fc46cff-hxmrl"]
Sep 15 19:03:18.233889 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:18.233855    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b982c2f-7303-4ac6-a4e0-82068d51864b" path="/var/lib/kubelet/pods/9b982c2f-7303-4ac6-a4e0-82068d51864b/volumes"
Sep 15 19:03:18.234249 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:18.234070    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec64721e-1809-4646-ac17-1c840891f3ba" path="/var/lib/kubelet/pods/ec64721e-1809-4646-ac17-1c840891f3ba/volumes"
Sep 15 19:03:21.901674 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:21.901638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:21.968957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:21.968927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:25.699847 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:25.699810    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:25.797875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:25.797845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:57.631258 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:57.631221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:59.050170 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:59.050122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:59.165450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:59.165416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:59.414226 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:59.414144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:59.728829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:59.728749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:03:59.806809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:03:59.806771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:04:05.748549 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:05.748517    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:04:05.851973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:05.851944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:04:09.680849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.680818    2584 generic.go:356] "Generic (PLEG): container finished" podID="0eb5d451-4682-4cf9-ac47-60323346ce64" containerID="060b4b52df1987d2853eb9ff5686ef9a1e8339f32134c76fca8a7862386de2d8" exitCode=0
Sep 15 19:04:09.681300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.680879    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" event={"ID":"0eb5d451-4682-4cf9-ac47-60323346ce64","Type":"ContainerDied","Data":"060b4b52df1987d2853eb9ff5686ef9a1e8339f32134c76fca8a7862386de2d8"}
Sep 15 19:04:09.681300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.681231    2584 scope.go:117] "RemoveContainer" containerID="060b4b52df1987d2853eb9ff5686ef9a1e8339f32134c76fca8a7862386de2d8"
Sep 15 19:04:09.682197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.682164    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:04:09.682291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.682233    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:04:09.682291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.682267    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:04:09.970986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:09.970906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:04:10.072901 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:10.072872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:04:10.687870 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:10.687831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" event={"ID":"0eb5d451-4682-4cf9-ac47-60323346ce64","Type":"ContainerStarted","Data":"bb97c346764ad777585b22ea7a31fd231829cfaa6c1a5a987026ebf78f45cfe6"}
Sep 15 19:04:11.609093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:11.609058    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:04:11.610216 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:11.610189    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zp9s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 15 19:04:11.610327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:11.610244    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" podUID="0eb5d451-4682-4cf9-ac47-60323346ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 15 19:04:12.609521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:12.609483    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zp9s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 15 19:04:12.609906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:12.609540    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" podUID="0eb5d451-4682-4cf9-ac47-60323346ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 15 19:04:13.609490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:13.609451    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zp9s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 15 19:04:13.609686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:13.609508    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" podUID="0eb5d451-4682-4cf9-ac47-60323346ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 15 19:04:14.609509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:14.609469    2584 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zp9s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 15 19:04:14.609718 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:14.609526    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8" podUID="0eb5d451-4682-4cf9-ac47-60323346ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 15 19:04:15.098060 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:15.098025    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:04:15.612522 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:15.612490    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:04:15.715943 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:15.715911    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zp9s8"
Sep 15 19:04:32.934683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:32.934627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:04:50.015680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:04:50.015637    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:03.581733 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.581697    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"]
Sep 15 19:05:03.582146 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.582072    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec64721e-1809-4646-ac17-1c840891f3ba" containerName="authorino"
Sep 15 19:05:03.582146 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.582084    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec64721e-1809-4646-ac17-1c840891f3ba" containerName="authorino"
Sep 15 19:05:03.582146 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.582140    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec64721e-1809-4646-ac17-1c840891f3ba" containerName="authorino"
Sep 15 19:05:03.584237 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.584221    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.586669 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.586647    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 19:05:03.587423 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.587407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 19:05:03.587575 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.587558    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bttfv\""
Sep 15 19:05:03.587649 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.587560    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 19:05:03.595380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.595355    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"]
Sep 15 19:05:03.682320 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.682289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:03.706538 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.706505    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.706707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.706543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvw7\" (UniqueName: \"kubernetes.io/projected/e91ec43c-68eb-4048-bc39-19f686e18108-kube-api-access-hbvw7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.706707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.706635    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.706707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.706680    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.706823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.706710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91ec43c-68eb-4048-bc39-19f686e18108-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.706823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.706749    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.807978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.807933    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808036    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvw7\" (UniqueName: \"kubernetes.io/projected/e91ec43c-68eb-4048-bc39-19f686e18108-kube-api-access-hbvw7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808154    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91ec43c-68eb-4048-bc39-19f686e18108-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808399    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.808563 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.808530    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.810443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.810417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91ec43c-68eb-4048-bc39-19f686e18108-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.810672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.810654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91ec43c-68eb-4048-bc39-19f686e18108-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.815473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.815451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvw7\" (UniqueName: \"kubernetes.io/projected/e91ec43c-68eb-4048-bc39-19f686e18108-kube-api-access-hbvw7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb\" (UID: \"e91ec43c-68eb-4048-bc39-19f686e18108\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:03.895690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:03.895574    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:04.170021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:04.169993    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"]
Sep 15 19:05:04.171356 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:05:04.171329    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91ec43c_68eb_4048_bc39_19f686e18108.slice/crio-1b133ad40a50c24356de63182031546b547966049f210078dc767375f9635365 WatchSource:0}: Error finding container 1b133ad40a50c24356de63182031546b547966049f210078dc767375f9635365: Status 404 returned error can't find the container with id 1b133ad40a50c24356de63182031546b547966049f210078dc767375f9635365
Sep 15 19:05:04.988013 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:04.987976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb" event={"ID":"e91ec43c-68eb-4048-bc39-19f686e18108","Type":"ContainerStarted","Data":"1b133ad40a50c24356de63182031546b547966049f210078dc767375f9635365"}
Sep 15 19:05:05.995355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:05.995319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb" event={"ID":"e91ec43c-68eb-4048-bc39-19f686e18108","Type":"ContainerStarted","Data":"8cf7e4bf9b532993414e45a8e42a5c2fbc21fdd1986ba7fd2ce570272c50be72"}
Sep 15 19:05:06.012086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:06.012037    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb" podStartSLOduration=1.850628451 podStartE2EDuration="3.012021515s" totalImagesPullingTime="1.161393064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:06.010887516 +0000 UTC m=+858.165255984" watchObservedRunningTime="2026-09-15 19:05:06.012021515 +0000 UTC m=+858.166389983"
Sep 15 19:05:12.381037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.381001    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"]
Sep 15 19:05:12.383599 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.383577    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.385960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.385935    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 19:05:12.398830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.398803    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"]
Sep 15 19:05:12.488329 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.488293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.488507 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.488349    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.488507 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.488462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88af5818-97c1-4827-855e-bb41d6e94254-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.488590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.488512    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.488590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.488530    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzps\" (UniqueName: \"kubernetes.io/projected/88af5818-97c1-4827-855e-bb41d6e94254-kube-api-access-kzzps\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.488590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.488555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.529046 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.529012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:12.589168 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589191    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88af5818-97c1-4827-855e-bb41d6e94254-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589218    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589236    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzps\" (UniqueName: \"kubernetes.io/projected/88af5818-97c1-4827-855e-bb41d6e94254-kube-api-access-kzzps\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589598 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589370    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589751 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589721    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.589751 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.589739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.591620 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.591587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88af5818-97c1-4827-855e-bb41d6e94254-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.591776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.591759    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88af5818-97c1-4827-855e-bb41d6e94254-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.596999 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.596972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzps\" (UniqueName: \"kubernetes.io/projected/88af5818-97c1-4827-855e-bb41d6e94254-kube-api-access-kzzps\") pod \"e2e-trlp-test-simulated-kserve-598466548d-69z2m\" (UID: \"88af5818-97c1-4827-855e-bb41d6e94254\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.696770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.696674    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:12.969767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:12.969740    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"]
Sep 15 19:05:12.974488 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:05:12.974455    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88af5818_97c1_4827_855e_bb41d6e94254.slice/crio-395166b08c5a56bb31bfb4082968cf32ede6baaabd0b5f8b26e20a254efcc592 WatchSource:0}: Error finding container 395166b08c5a56bb31bfb4082968cf32ede6baaabd0b5f8b26e20a254efcc592: Status 404 returned error can't find the container with id 395166b08c5a56bb31bfb4082968cf32ede6baaabd0b5f8b26e20a254efcc592
Sep 15 19:05:13.035289 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:13.035255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m" event={"ID":"88af5818-97c1-4827-855e-bb41d6e94254","Type":"ContainerStarted","Data":"395166b08c5a56bb31bfb4082968cf32ede6baaabd0b5f8b26e20a254efcc592"}
Sep 15 19:05:13.896233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:13.896197    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:13.896738 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:13.896266    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:13.915469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:13.915425    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:14.044447 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:14.044401    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m" event={"ID":"88af5818-97c1-4827-855e-bb41d6e94254","Type":"ContainerStarted","Data":"ef6756b51e545217de64b60ad6a26e29d7430dbebc24a166f6f1db74bb42bbb8"}
Sep 15 19:05:14.056492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:14.056454    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nqqwb"
Sep 15 19:05:14.065995 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:14.065940    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m" podStartSLOduration=1.844644137 podStartE2EDuration="2.065919325s" totalImagesPullingTime="221.275188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:14.063722961 +0000 UTC m=+866.218091426" watchObservedRunningTime="2026-09-15 19:05:14.065919325 +0000 UTC m=+866.220287792"
Sep 15 19:05:22.697607 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:22.697561    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:22.697607 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:22.697644    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:22.710256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:22.710230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:23.106719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:23.106689    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-69z2m"
Sep 15 19:05:27.211526 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:27.211486    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:32.633956 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:32.633917    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:35.480272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:35.480236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:43.086511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.086431    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"]
Sep 15 19:05:43.089422 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.089397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.091904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.091885    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 19:05:43.100518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.100490    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"]
Sep 15 19:05:43.112719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.112687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:05:43.183635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.183580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.183776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.183682    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.183776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.183710    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8sdg\" (UniqueName: \"kubernetes.io/projected/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-kube-api-access-q8sdg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.183776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.183741    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.183909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.183797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.183945 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.183906    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.284835 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.284789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.284853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.284913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.284947    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8sdg\" (UniqueName: \"kubernetes.io/projected/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-kube-api-access-q8sdg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.284982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.285004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.285260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285566 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.285545    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.285655 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.285599    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.287338 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.287316    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.287640 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.287598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.293077 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.293049    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8sdg\" (UniqueName: \"kubernetes.io/projected/e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c-kube-api-access-q8sdg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-586bd\" (UID: \"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.420606 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.420505    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:43.705007 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:43.704976    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"]
Sep 15 19:05:43.707481 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:05:43.707447    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode97dc8aa_b696_4c93_b4ec_73a9e8fd9a5c.slice/crio-1c395f49b89a1b89fe3d0202347052ed3f4208f08b3a2f240ae0dd76d4a173a2 WatchSource:0}: Error finding container 1c395f49b89a1b89fe3d0202347052ed3f4208f08b3a2f240ae0dd76d4a173a2: Status 404 returned error can't find the container with id 1c395f49b89a1b89fe3d0202347052ed3f4208f08b3a2f240ae0dd76d4a173a2
Sep 15 19:05:44.234171 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:44.234135    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd" event={"ID":"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c","Type":"ContainerStarted","Data":"8f8ede27d0f2b2fb9532b0c70a341ab00cbf665f0e50a8bed8f6f4e17fd516ae"}
Sep 15 19:05:44.234171 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:44.234175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd" event={"ID":"e97dc8aa-b696-4c93-b4ec-73a9e8fd9a5c","Type":"ContainerStarted","Data":"1c395f49b89a1b89fe3d0202347052ed3f4208f08b3a2f240ae0dd76d4a173a2"}
Sep 15 19:05:44.246077 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:44.246025    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd" podStartSLOduration=1.074524297 podStartE2EDuration="1.246008912s" totalImagesPullingTime="171.484615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:05:44.245073824 +0000 UTC m=+896.399442288" watchObservedRunningTime="2026-09-15 19:05:44.246008912 +0000 UTC m=+896.400377380"
Sep 15 19:05:53.421169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:53.421130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:53.421583 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:53.421193    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:53.433721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:53.433694    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:05:54.299204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:05:54.299174    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-586bd"
Sep 15 19:06:40.834669 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:40.834627    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cb7d44c65-6h2jr"]
Sep 15 19:06:40.835311 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:40.834849    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr" podUID="2331c363-cccd-4129-a5e2-72e6d4d16ad8" containerName="authorino" containerID="cri-o://10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054" gracePeriod=30
Sep 15 19:06:41.085192 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.085130    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:06:41.172802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.172772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:06:41.210707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.210673    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2331c363-cccd-4129-a5e2-72e6d4d16ad8-tls-cert\") pod \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") "
Sep 15 19:06:41.210871 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.210804    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt5n4\" (UniqueName: \"kubernetes.io/projected/2331c363-cccd-4129-a5e2-72e6d4d16ad8-kube-api-access-pt5n4\") pod \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\" (UID: \"2331c363-cccd-4129-a5e2-72e6d4d16ad8\") "
Sep 15 19:06:41.212802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.212771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2331c363-cccd-4129-a5e2-72e6d4d16ad8-kube-api-access-pt5n4" (OuterVolumeSpecName: "kube-api-access-pt5n4") pod "2331c363-cccd-4129-a5e2-72e6d4d16ad8" (UID: "2331c363-cccd-4129-a5e2-72e6d4d16ad8"). InnerVolumeSpecName "kube-api-access-pt5n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:06:41.223319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.223275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2331c363-cccd-4129-a5e2-72e6d4d16ad8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2331c363-cccd-4129-a5e2-72e6d4d16ad8" (UID: "2331c363-cccd-4129-a5e2-72e6d4d16ad8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:06:41.311687 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.311644    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5n4\" (UniqueName: \"kubernetes.io/projected/2331c363-cccd-4129-a5e2-72e6d4d16ad8-kube-api-access-pt5n4\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:06:41.311687 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.311679    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2331c363-cccd-4129-a5e2-72e6d4d16ad8-tls-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:06:41.575450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.575416    2584 generic.go:356] "Generic (PLEG): container finished" podID="2331c363-cccd-4129-a5e2-72e6d4d16ad8" containerID="10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054" exitCode=0
Sep 15 19:06:41.575635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.575466    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr"
Sep 15 19:06:41.575635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.575507    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr" event={"ID":"2331c363-cccd-4129-a5e2-72e6d4d16ad8","Type":"ContainerDied","Data":"10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054"}
Sep 15 19:06:41.575635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.575559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb7d44c65-6h2jr" event={"ID":"2331c363-cccd-4129-a5e2-72e6d4d16ad8","Type":"ContainerDied","Data":"68594e6e14b66e3bcd7f6b6bd91e4bd75c01a6d3b78ed21f1fce304fb14104e0"}
Sep 15 19:06:41.575635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.575583    2584 scope.go:117] "RemoveContainer" containerID="10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054"
Sep 15 19:06:41.586507 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.586491    2584 scope.go:117] "RemoveContainer" containerID="10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054"
Sep 15 19:06:41.586928 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:06:41.586907    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054\": container with ID starting with 10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054 not found: ID does not exist" containerID="10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054"
Sep 15 19:06:41.587024 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.586937    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054"} err="failed to get container status \"10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054\": rpc error: code = NotFound desc = could not find container \"10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054\": container with ID starting with 10ea7e4238f190d921d427b438f7680492860552c304e89040d3a77c9ab71054 not found: ID does not exist"
Sep 15 19:06:41.588571 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.588547    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:06:41.595987 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.595965    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cb7d44c65-6h2jr"]
Sep 15 19:06:41.601295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.601276    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cb7d44c65-6h2jr"]
Sep 15 19:06:41.720902 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:41.720869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:06:42.233204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:06:42.233173    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2331c363-cccd-4129-a5e2-72e6d4d16ad8" path="/var/lib/kubelet/pods/2331c363-cccd-4129-a5e2-72e6d4d16ad8/volumes"
Sep 15 19:07:00.914885 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:00.914842    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:01.013597 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:01.013558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:19.794371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:19.794337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:26.205050 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:26.205017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:26.521715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:26.521685    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:27.369458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:27.369407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:27.827176 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:27.827133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:28.400840 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:28.400803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:28.490446 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:28.490393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:35.954973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:35.954936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.040197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.040134    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.140702 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.140665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.375108 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.375028    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"]
Sep 15 19:07:43.375588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.375573    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2331c363-cccd-4129-a5e2-72e6d4d16ad8" containerName="authorino"
Sep 15 19:07:43.375672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.375592    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2331c363-cccd-4129-a5e2-72e6d4d16ad8" containerName="authorino"
Sep 15 19:07:43.375748 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.375736    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2331c363-cccd-4129-a5e2-72e6d4d16ad8" containerName="authorino"
Sep 15 19:07:43.378004 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.377984    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.380533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.380509    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-900c88-openshift-default-dockercfg-bz27g\""
Sep 15 19:07:43.380892 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.380864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-socket\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.380983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.380910    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddvv8\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-kube-api-access-ddvv8\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.380983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.380941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-envoy\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.380983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.380967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-data\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.381112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.380998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45676d20-747e-46f4-8ef5-d3a9dd784777-istiod-ca-cert\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.381112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.381024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-token\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.381186 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.381111    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-podinfo\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.381186 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.381160    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-credential-socket\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.381255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.381203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-certs\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.399894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.399864    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"]
Sep 15 19:07:43.410448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.410411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.481983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.481943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-socket\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.481997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddvv8\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-kube-api-access-ddvv8\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482039    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-envoy\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482064    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-data\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45676d20-747e-46f4-8ef5-d3a9dd784777-istiod-ca-cert\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-token\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-podinfo\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-credential-socket\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-certs\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.482530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.482390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-socket\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.483530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.483441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-credential-socket\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.483972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.483950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45676d20-747e-46f4-8ef5-d3a9dd784777-istiod-ca-cert\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.488061 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.487986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-data\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.488668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.488331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-envoy\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.489089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.489068    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-certs\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.491699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.491358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-podinfo\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.497561 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.496089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddvv8\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-kube-api-access-ddvv8\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.503795 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.503767    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-token\") pod \"e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") " pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.507377 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.507353    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"]
Sep 15 19:07:43.511154 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.511133    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.514100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.514081    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f0ad35-openshift-default-dockercfg-d8vcr\""
Sep 15 19:07:43.523799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.523233    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"]
Sep 15 19:07:43.543578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.541998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.564041 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.564009    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"]
Sep 15 19:07:43.567337 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.567313    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.569627 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.569592    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9de44d-openshift-default-dockercfg-sxms8\""
Sep 15 19:07:43.580223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.580199    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"]
Sep 15 19:07:43.583924 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.583893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646a529b-e291-4e71-b761-c838288e91e4-istiod-ca-cert\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584061 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.583956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-data\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-socket\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-envoy\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2jk\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-kube-api-access-nv2jk\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-data\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584183    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkt46\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-kube-api-access-lkt46\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istiod-ca-cert\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584283    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-certs\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-podinfo\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584351    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-socket\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-certs\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584407    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646a529b-e291-4e71-b761-c838288e91e4-istio-podinfo\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584434    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-credential-socket\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-token\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.584788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584514    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-istio-token\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-credential-socket\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.584788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.584588    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-envoy\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.640129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.640036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.685703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-data\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.685703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685703    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkt46\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-kube-api-access-lkt46\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istiod-ca-cert\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-certs\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-podinfo\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685872    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-socket\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-certs\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646a529b-e291-4e71-b761-c838288e91e4-istio-podinfo\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.685966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685956    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-credential-socket\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.685982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-token\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686015    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-istio-token\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-credential-socket\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-envoy\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646a529b-e291-4e71-b761-c838288e91e4-istiod-ca-cert\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-data\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.686233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-socket\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.686575 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-envoy\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.686575 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.686324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2jk\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-kube-api-access-nv2jk\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.687286 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.687246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-data\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.687404 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.687305    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-certs\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.687569 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.687535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646a529b-e291-4e71-b761-c838288e91e4-istiod-ca-cert\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.687702 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.687606    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-socket\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.687927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.687901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-data\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.687990    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istiod-ca-cert\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.688210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-credential-socket\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.688227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-credential-socket\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.688540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-certs\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.689334    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.692145    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-socket\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.696417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2jk\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-kube-api-access-nv2jk\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.696673    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-envoy\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.697844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.697502    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646a529b-e291-4e71-b761-c838288e91e4-istio-podinfo\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.698888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.698072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-envoy\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.698888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.698473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-istio-token\") pod \"e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") " pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.699310 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.699264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-token\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.699690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.699649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-podinfo\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.700214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.700192    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkt46\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-kube-api-access-lkt46\") pod \"e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") " pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.838937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.838854    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:43.882908 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.882872    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:43.906771 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.902169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.994976 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.994947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:43.995389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.995361    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"]
Sep 15 19:07:43.995623 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:07:43.995587    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45676d20_747e_46f4_8ef5_d3a9dd784777.slice/crio-d59ea3ccf9722dcdc9857798abb7c5c7880ade226d151a17076c52b736d9865a WatchSource:0}: Error finding container d59ea3ccf9722dcdc9857798abb7c5c7880ade226d151a17076c52b736d9865a: Status 404 returned error can't find the container with id d59ea3ccf9722dcdc9857798abb7c5c7880ade226d151a17076c52b736d9865a
Sep 15 19:07:43.997991 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.997958    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:43.998107 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.998031    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:43.998107 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:43.998075    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.129054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.129025    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"]
Sep 15 19:07:44.130736 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:07:44.130710    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e57c12e_68ef_4451_9be0_7a4a9576c8d0.slice/crio-ac2a6dca6e9ddf944a604589deb5bfebae3020aeb4cc6b11458d1a95fd825108 WatchSource:0}: Error finding container ac2a6dca6e9ddf944a604589deb5bfebae3020aeb4cc6b11458d1a95fd825108: Status 404 returned error can't find the container with id ac2a6dca6e9ddf944a604589deb5bfebae3020aeb4cc6b11458d1a95fd825108
Sep 15 19:07:44.133249 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.133212    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.133371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.133298    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.133371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.133343    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.186631 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.186592    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"]
Sep 15 19:07:44.188788 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:07:44.188757    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646a529b_e291_4e71_b761_c838288e91e4.slice/crio-6dbce975f2bdb0909a995c12054e883344cc4764d1e65d0f1f864cf57e00f60c WatchSource:0}: Error finding container 6dbce975f2bdb0909a995c12054e883344cc4764d1e65d0f1f864cf57e00f60c: Status 404 returned error can't find the container with id 6dbce975f2bdb0909a995c12054e883344cc4764d1e65d0f1f864cf57e00f60c
Sep 15 19:07:44.191076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.191040    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.191183 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.191133    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.191250 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.191181    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:44.378939 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.378906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:44.480235 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.480138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:44.802995 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.802958    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:44.944779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.944744    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" event={"ID":"646a529b-e291-4e71-b761-c838288e91e4","Type":"ContainerStarted","Data":"0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"}
Sep 15 19:07:44.944952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.944806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" event={"ID":"646a529b-e291-4e71-b761-c838288e91e4","Type":"ContainerStarted","Data":"6dbce975f2bdb0909a995c12054e883344cc4764d1e65d0f1f864cf57e00f60c"}
Sep 15 19:07:44.946566 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.946539    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" event={"ID":"5e57c12e-68ef-4451-9be0-7a4a9576c8d0","Type":"ContainerStarted","Data":"47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"}
Sep 15 19:07:44.946730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.946631    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" event={"ID":"5e57c12e-68ef-4451-9be0-7a4a9576c8d0","Type":"ContainerStarted","Data":"ac2a6dca6e9ddf944a604589deb5bfebae3020aeb4cc6b11458d1a95fd825108"}
Sep 15 19:07:44.948315 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.948280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" event={"ID":"45676d20-747e-46f4-8ef5-d3a9dd784777","Type":"ContainerStarted","Data":"b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"}
Sep 15 19:07:44.948437 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.948327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" event={"ID":"45676d20-747e-46f4-8ef5-d3a9dd784777","Type":"ContainerStarted","Data":"d59ea3ccf9722dcdc9857798abb7c5c7880ade226d151a17076c52b736d9865a"}
Sep 15 19:07:44.964958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.964875    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podStartSLOduration=1.96485945 podStartE2EDuration="1.96485945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:44.96312731 +0000 UTC m=+1017.117495778" watchObservedRunningTime="2026-09-15 19:07:44.96485945 +0000 UTC m=+1017.119227918"
Sep 15 19:07:44.983355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:44.983307    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podStartSLOduration=1.9832908809999998 podStartE2EDuration="1.983290881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:44.981341263 +0000 UTC m=+1017.135709721" watchObservedRunningTime="2026-09-15 19:07:44.983290881 +0000 UTC m=+1017.137659349"
Sep 15 19:07:45.001968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.001910    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podStartSLOduration=2.001891027 podStartE2EDuration="2.001891027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:44.998813452 +0000 UTC m=+1017.153181945" watchObservedRunningTime="2026-09-15 19:07:45.001891027 +0000 UTC m=+1017.156259495"
Sep 15 19:07:45.689692 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.689653    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:45.694841 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.694817    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:45.839486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.839450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:45.844541 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.844513    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:45.883567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.883539    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:45.888352 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.888327    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:45.955348 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.955261    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:45.955348 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.955314    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:45.955348 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.955339    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:45.956048 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.956024    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:07:45.956154 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.956092    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:07:45.956234 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:45.956219    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:07:46.928405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:46.928366    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:47.060500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:47.060469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:47.711308 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:47.711262    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:47.762327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:47.762289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:48.195271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:48.195239    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:48.300790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:48.300758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:48.400495 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:48.400465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:48.776229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:48.776200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:48.864944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:48.864903    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:49.149362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.149289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:49.217061 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.217028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:49.480587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.480504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:49.759492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.759452    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"]
Sep 15 19:07:49.762014 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.761993    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.764276 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.764247    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f0ad35\""
Sep 15 19:07:49.764413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.764347    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f0ad35-dockercfg-mvmcr\""
Sep 15 19:07:49.773952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.773927    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"]
Sep 15 19:07:49.854947 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.854917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7bm\" (UniqueName: \"kubernetes.io/projected/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-kube-api-access-4p7bm\") pod \"payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.855142 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.854952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.955963 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.955927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7bm\" (UniqueName: \"kubernetes.io/projected/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-kube-api-access-4p7bm\") pod \"payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.955963 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.955963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.956734 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.956716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.968449 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.968419    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7bm\" (UniqueName: \"kubernetes.io/projected/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-kube-api-access-4p7bm\") pod \"payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:49.993575 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:49.993531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.073898 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.073797    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:50.083438 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.083396    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.220874 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.220831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.283815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.283783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.350547 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.350514    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"]
Sep 15 19:07:50.351599 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:07:50.351573    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5889c3_c6bf_4403_a52d_35c497b0ebe9.slice/crio-38fd135b8bd219bdf537f00dab7662861261572b80bec1072a6f0b33078610ed WatchSource:0}: Error finding container 38fd135b8bd219bdf537f00dab7662861261572b80bec1072a6f0b33078610ed: Status 404 returned error can't find the container with id 38fd135b8bd219bdf537f00dab7662861261572b80bec1072a6f0b33078610ed
Sep 15 19:07:50.353630 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.353599    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:07:50.675427 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.675331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.796275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.796227    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.880265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.880228    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:50.996534 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:50.996436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn" event={"ID":"ef5889c3-c6bf-4403-a52d-35c497b0ebe9","Type":"ContainerStarted","Data":"38fd135b8bd219bdf537f00dab7662861261572b80bec1072a6f0b33078610ed"}
Sep 15 19:07:51.240241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:51.240188    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:51.494302 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:51.494274    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:51.560028 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:51.559991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:51.705280 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:51.705189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:52.001640 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.001526    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"]
Sep 15 19:07:52.005358 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.005324    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.005502 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.005416    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn" event={"ID":"ef5889c3-c6bf-4403-a52d-35c497b0ebe9","Type":"ContainerStarted","Data":"cf58bb312293b68e3037dba28a4b54f652da7ea863ab35550324b969c80a31d5"}
Sep 15 19:07:52.011398 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.011377    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a7a10c67-openshift-default-dockercfg-j946l\""
Sep 15 19:07:52.029580 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.029543    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"]
Sep 15 19:07:52.077273 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-credential-socket\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-podinfo\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-data\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istiod-ca-cert\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077451    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2k9\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-kube-api-access-rs2k9\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077594    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-token\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-envoy\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-socket\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.077827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.077752    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-certs\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.102335 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.102303    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:52.111056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.111011    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn" podStartSLOduration=2.01968255 podStartE2EDuration="3.110998453s" totalImagesPullingTime="1.091315903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:52.110653775 +0000 UTC m=+1024.265022243" watchObservedRunningTime="2026-09-15 19:07:52.110998453 +0000 UTC m=+1024.265366920"
Sep 15 19:07:52.178483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-credential-socket\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-podinfo\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-data\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178569    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istiod-ca-cert\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2k9\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-kube-api-access-rs2k9\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-token\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-envoy\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178740    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-socket\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.178877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-certs\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.179076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.178990    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-credential-socket\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.179076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.179057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-data\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.179186 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.179077    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-certs\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.179369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.179345    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istiod-ca-cert\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.179369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.179360    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-socket\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.181191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.181163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-envoy\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.181401 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.181382    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-podinfo\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.188737 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.188713    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2k9\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-kube-api-access-rs2k9\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.189017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.188997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-token\") pod \"e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") " pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.225665 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.225601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:52.318209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.318173    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:52.594558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.594528    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"]
Sep 15 19:07:52.595858 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:07:52.595833    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d6565a_d85e_4c08_97c7_1fc61d09cdc0.slice/crio-22d7df445ba4b4ff716f45b718aae70313ac223316ce0bdb43762b001fd5b9c0 WatchSource:0}: Error finding container 22d7df445ba4b4ff716f45b718aae70313ac223316ce0bdb43762b001fd5b9c0: Status 404 returned error can't find the container with id 22d7df445ba4b4ff716f45b718aae70313ac223316ce0bdb43762b001fd5b9c0
Sep 15 19:07:52.599587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.599537    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:52.599703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.599686    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:52.599752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.599736    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:07:52.829995 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:52.829957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:53.021037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:53.020991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" event={"ID":"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0","Type":"ContainerStarted","Data":"a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"}
Sep 15 19:07:53.021247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:53.021092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" event={"ID":"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0","Type":"ContainerStarted","Data":"22d7df445ba4b4ff716f45b718aae70313ac223316ce0bdb43762b001fd5b9c0"}
Sep 15 19:07:53.021247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:53.021159    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:07:53.041573 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:53.041511    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podStartSLOduration=2.041494059 podStartE2EDuration="2.041494059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:07:53.03844247 +0000 UTC m=+1025.192810950" watchObservedRunningTime="2026-09-15 19:07:53.041494059 +0000 UTC m=+1025.195862527"
Sep 15 19:07:53.318927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:53.318841    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:53.325244 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:53.325213    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:54.029709 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:54.029672    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:54.030649 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:54.030623    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:07:54.279383 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:54.279343    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:54.485297 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:54.485199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:54.581587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:54.581548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:54.830851 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:54.830781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:55.522753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:55.522713    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:56.425705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:56.425668    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:57.153638 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:57.147206    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:57.215253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:57.215223    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:57.701389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:57.701296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:07:57.793071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:07:57.793031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:01.572598 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.572558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:01.894351 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.894255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:01.953244 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.953208    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:01.996809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.996766    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 19:08:01.997015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.996835    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 19:08:01.997015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.996848    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 19:08:01.997015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.996909    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 19:08:01.997015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.996832    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 19:08:01.997015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:01.996982    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 19:08:04.032515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.032455    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:08:04.123849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.123760    2584 generic.go:356] "Generic (PLEG): container finished" podID="646a529b-e291-4e71-b761-c838288e91e4" containerID="0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b" exitCode=0
Sep 15 19:08:04.123849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.123830    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" event={"ID":"646a529b-e291-4e71-b761-c838288e91e4","Type":"ContainerDied","Data":"0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"}
Sep 15 19:08:04.124335 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.124307    2584 scope.go:117] "RemoveContainer" containerID="0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"
Sep 15 19:08:04.125670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.125632    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:04.125808 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.125729    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:04.125808 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.125779    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:04.126410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.126385    2584 generic.go:356] "Generic (PLEG): container finished" podID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerID="b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2" exitCode=0
Sep 15 19:08:04.126525 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.126440    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" event={"ID":"45676d20-747e-46f4-8ef5-d3a9dd784777","Type":"ContainerDied","Data":"b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"}
Sep 15 19:08:04.127078 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.127025    2584 scope.go:117] "RemoveContainer" containerID="b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"
Sep 15 19:08:04.128270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.128237    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:04.128367 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.128319    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:04.128422 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:04.128366    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:05.139167 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.139121    2584 generic.go:356] "Generic (PLEG): container finished" podID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerID="47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae" exitCode=0
Sep 15 19:08:05.139708 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.139213    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" event={"ID":"5e57c12e-68ef-4451-9be0-7a4a9576c8d0","Type":"ContainerDied","Data":"47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"}
Sep 15 19:08:05.140586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.140535    2584 scope.go:117] "RemoveContainer" containerID="47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"
Sep 15 19:08:05.142000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.141964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" event={"ID":"646a529b-e291-4e71-b761-c838288e91e4","Type":"ContainerStarted","Data":"21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a"}
Sep 15 19:08:05.144452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.144038    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:05.144452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.144136    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:05.144452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.144185    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:05.149893 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.147802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" event={"ID":"45676d20-747e-46f4-8ef5-d3a9dd784777","Type":"ContainerStarted","Data":"8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d"}
Sep 15 19:08:05.690787 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.690744    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:08:05.690980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.690888    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 15 19:08:05.690980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.690934    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 15 19:08:05.885491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.885053    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:08:05.885491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.885186    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 15 19:08:05.885491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:05.885232    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 15 19:08:06.165964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.165930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" event={"ID":"5e57c12e-68ef-4451-9be0-7a4a9576c8d0","Type":"ContainerStarted","Data":"f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a"}
Sep 15 19:08:06.689991 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.689943    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 15 19:08:06.690215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.690013    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 15 19:08:06.839311 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.839277    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:08:06.839714 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.839603    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:06.839905 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.839881    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:06.884752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.884654    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 15 19:08:06.885199 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:06.885155    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 15 19:08:07.690937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.690883    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 15 19:08:07.691463 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.690973    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 15 19:08:07.762377 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.762332    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:07.839438 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.839395    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:07.839660 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.839467    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:07.850766 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.850727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:07.883503 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.883467    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 15 19:08:07.883834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.883786    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 15 19:08:07.958982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:07.958829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:08.122199 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.122103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:08.690027 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.689978    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 15 19:08:08.690242 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.690067    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 15 19:08:08.839800 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.839746    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:08.840273 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.839826    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:08.883708 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.883657    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 15 19:08:08.883885 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:08.883757    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 15 19:08:09.147022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.146959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:09.690697 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.690654    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 15 19:08:09.690908 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.690727    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 15 19:08:09.839776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.839731    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:09.840000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.839804    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:09.884267 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.884222    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 15 19:08:09.884475 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:09.884313    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 15 19:08:10.690250 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:10.690213    2584 patch_prober.go:28] interesting pod/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 15 19:08:10.690556 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:10.690527    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 15 19:08:10.839468 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:10.839428    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:10.839788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:10.839758    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:10.884332 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:10.884255    2584 patch_prober.go:28] interesting pod/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 15 19:08:10.885015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:10.884958    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 15 19:08:11.014442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.014402    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:11.424656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.424524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:11.693823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.693752    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:08:11.694116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.694097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:08:11.694744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.694721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:08:11.780187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.780153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:11.840078 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.840032    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:11.840278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.840114    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:11.879906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.879865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:11.887763 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.887736    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:08:11.888213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.888005    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:08:11.888778 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:11.888758    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:08:12.081791 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.081760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:12.216094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.216042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:12.321917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.321872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:12.383283 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.383208    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"]
Sep 15 19:08:12.386744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.386720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.390231 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.390204    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3daa58-openshift-default-dockercfg-5m2s4\""
Sep 15 19:08:12.399147 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.399113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"]
Sep 15 19:08:12.417623 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.417587    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:12.508934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.508774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-credential-socket\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.508934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.508815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-podinfo\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.508934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.508854    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-socket\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.508934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.508880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48z9\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-kube-api-access-z48z9\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.508934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.508906    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983248b6-b680-4c9e-bb1e-bc22ae769201-istiod-ca-cert\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.509240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.508990    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-envoy\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.509240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.509069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-token\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.509240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.509116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-certs\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.509240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.509165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-data\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.520997 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.520816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:12.610521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983248b6-b680-4c9e-bb1e-bc22ae769201-istiod-ca-cert\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-envoy\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610595    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-token\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-certs\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-data\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610780    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-credential-socket\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-podinfo\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-socket\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.610894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.610865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48z9\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-kube-api-access-z48z9\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.611152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.611079    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-certs\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.611288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.611258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-credential-socket\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.611435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.611301    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983248b6-b680-4c9e-bb1e-bc22ae769201-istiod-ca-cert\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.611435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.611305    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-data\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.611530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.611513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-socket\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.613156 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.613133    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-envoy\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.613429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.613408    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-podinfo\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.622676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.622646    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48z9\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-kube-api-access-z48z9\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.624008 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.623979    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-token\") pod \"e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") " pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.702857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.702769    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:12.851799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.840072    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 19:08:12.851799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:12.840134    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 19:08:13.034634 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.033866    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"]
Sep 15 19:08:13.053444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.053304    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:13.053444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.053410    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:13.053676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.053455    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:13.075060 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.075031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:13.233838 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.233793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" event={"ID":"983248b6-b680-4c9e-bb1e-bc22ae769201","Type":"ContainerStarted","Data":"701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"}
Sep 15 19:08:13.234032 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.233866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" event={"ID":"983248b6-b680-4c9e-bb1e-bc22ae769201","Type":"ContainerStarted","Data":"97bc1070f7df347001c8ed9b94fcf5bae960d5e8edde7314dedde360b6507598"}
Sep 15 19:08:13.261429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.261328    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podStartSLOduration=1.261300901 podStartE2EDuration="1.261300901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:13.257155275 +0000 UTC m=+1045.411523769" watchObservedRunningTime="2026-09-15 19:08:13.261300901 +0000 UTC m=+1045.415669370"
Sep 15 19:08:13.561830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.561767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:13.703806 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.703765    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:13.844817 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.844733    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:08:13.845149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.845124    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:08:13.845850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:13.845831    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:08:14.327766 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.327724    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:14.531257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.531221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:14.626546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.626455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:14.710463 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.710430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:14.774776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.774731    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"]
Sep 15 19:08:14.778027 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.778004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:14.780213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.780187    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:08:14.780342 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.780193    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:08:14.785713 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.785673    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"]
Sep 15 19:08:14.837662 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.837630    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg\" (UID: \"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:14.938923 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.938818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg\" (UID: \"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:14.939467 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:14.939442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg\" (UID: \"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:15.092708 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:15.092669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:15.184392 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:15.184361    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:15.245037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:15.244945    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:15.246055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:15.246036    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:15.320837 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:15.320760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:15.378706 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:15.377979    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"]
Sep 15 19:08:15.392937 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:08:15.392903    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e521a9b_35d6_4a55_88b6_5b935c3a2d3b.slice/crio-ca550060f875fff9c64a5daaed339b3d2674edf6ee4ca43ed047f8b14aedee27 WatchSource:0}: Error finding container ca550060f875fff9c64a5daaed339b3d2674edf6ee4ca43ed047f8b14aedee27: Status 404 returned error can't find the container with id ca550060f875fff9c64a5daaed339b3d2674edf6ee4ca43ed047f8b14aedee27
Sep 15 19:08:16.021756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.021719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:16.251868 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.251835    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg" event={"ID":"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b","Type":"ContainerStarted","Data":"ca550060f875fff9c64a5daaed339b3d2674edf6ee4ca43ed047f8b14aedee27"}
Sep 15 19:08:16.253482 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.253453    2584 generic.go:356] "Generic (PLEG): container finished" podID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerID="a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea" exitCode=0
Sep 15 19:08:16.253649 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.253537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" event={"ID":"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0","Type":"ContainerDied","Data":"a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"}
Sep 15 19:08:16.254053 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.254036    2584 scope.go:117] "RemoveContainer" containerID="a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"
Sep 15 19:08:16.255239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.255202    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:16.255378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.255291    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:16.255378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.255341    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:16.783080 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.783041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:16.898945 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:16.898911    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:17.263751 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.263647    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" event={"ID":"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0","Type":"ContainerStarted","Data":"29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe"}
Sep 15 19:08:17.318650 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.318550    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:08:17.318869 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.318828    2584 patch_prober.go:28] interesting pod/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 19:08:17.318957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.318875    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 19:08:17.346179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.346083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:17.445214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.445169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:17.542689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.542158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:17.699593 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:17.699541    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:18.319401 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:18.319363    2584 patch_prober.go:28] interesting pod/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 19:08:18.319870 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:18.319440    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 19:08:18.413216 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:18.413183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:18.536636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:18.534369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:18.704920 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:18.704838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:19.059521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:19.059478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:19.162251 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:19.162220    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:19.319369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:19.319239    2584 patch_prober.go:28] interesting pod/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 19:08:19.319369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:19.319351    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 19:08:19.489354 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:19.489314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:20.284717 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:20.284675    2584 generic.go:356] "Generic (PLEG): container finished" podID="4e521a9b-35d6-4a55-88b6-5b935c3a2d3b" containerID="f54342b640c550d9a6b86b05571f5ab0c41a16f42e36230674fd2d046bda7efa" exitCode=0
Sep 15 19:08:20.284894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:20.284762    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg" event={"ID":"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b","Type":"ContainerDied","Data":"f54342b640c550d9a6b86b05571f5ab0c41a16f42e36230674fd2d046bda7efa"}
Sep 15 19:08:20.319410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:20.319370    2584 patch_prober.go:28] interesting pod/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 19:08:20.319547 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:20.319444    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 19:08:20.405551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:20.405519    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:21.049065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.049022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:21.319726 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.319190    2584 patch_prober.go:28] interesting pod/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 15 19:08:21.319726 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.319264    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 15 19:08:21.479529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.479505    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:21.616351 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.616258    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b-maas-api-service-ca\") pod \"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b\" (UID: \"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b\") "
Sep 15 19:08:21.616685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.616660    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e521a9b-35d6-4a55-88b6-5b935c3a2d3b" (UID: "4e521a9b-35d6-4a55-88b6-5b935c3a2d3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:08:21.692734 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.692696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:21.717600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:21.717565    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:22.301729 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:22.301697    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"
Sep 15 19:08:22.301913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:22.301699    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg" event={"ID":"4e521a9b-35d6-4a55-88b6-5b935c3a2d3b","Type":"ContainerDied","Data":"ca550060f875fff9c64a5daaed339b3d2674edf6ee4ca43ed047f8b14aedee27"}
Sep 15 19:08:22.301913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:22.301830    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca550060f875fff9c64a5daaed339b3d2674edf6ee4ca43ed047f8b14aedee27"
Sep 15 19:08:22.366960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:22.366929    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:08:22.367492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:22.367460    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:08:22.368529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:22.368510    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:08:24.910699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:24.910667    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:25.497669 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.497636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:25.599058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.599028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:25.699079 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.699049    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:25.713722 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.713683    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"]
Sep 15 19:08:25.716405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.716385    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7a10c67-fpfdg"]
Sep 15 19:08:25.792569 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.792532    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:25.847837 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.847800    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"]
Sep 15 19:08:25.848057 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.848030    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy" containerID="cri-o://29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe" gracePeriod=30
Sep 15 19:08:25.875483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:25.875446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:26.235897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.235810    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e521a9b-35d6-4a55-88b6-5b935c3a2d3b" path="/var/lib/kubelet/pods/4e521a9b-35d6-4a55-88b6-5b935c3a2d3b/volumes"
Sep 15 19:08:26.254819 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.254758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:26.353240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.353209    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:26.455298 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.455264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:26.551488 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.551449    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:26.824414 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.824322    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"]
Sep 15 19:08:26.825088 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.825056    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e521a9b-35d6-4a55-88b6-5b935c3a2d3b" containerName="revoke-keys"
Sep 15 19:08:26.825088 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.825086    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e521a9b-35d6-4a55-88b6-5b935c3a2d3b" containerName="revoke-keys"
Sep 15 19:08:26.825295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.825198    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e521a9b-35d6-4a55-88b6-5b935c3a2d3b" containerName="revoke-keys"
Sep 15 19:08:26.828135 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.828113    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.830633 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.830599    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8b287f5b-gw-openshift-default-dockercfg-62dwr\""
Sep 15 19:08:26.841652 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.841624    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"]
Sep 15 19:08:26.862828 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.862788    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:26.982125 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-certs\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-token\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-credential-socket\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-envoy\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-socket\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-podinfo\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4209a-caa5-43a1-a29f-5cefae17b39c-istiod-ca-cert\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982452    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97m2b\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-kube-api-access-97m2b\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.982560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.982521    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-data\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:26.993437 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:26.993407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:27.083996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.083898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97m2b\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-kube-api-access-97m2b\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.083996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.083950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-data\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-certs\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-token\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-credential-socket\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-envoy\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-socket\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-podinfo\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4209a-caa5-43a1-a29f-5cefae17b39c-istiod-ca-cert\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-data\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-socket\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084761 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-credential-socket\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084761 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084701    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-certs\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.084999 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.084979    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4209a-caa5-43a1-a29f-5cefae17b39c-istiod-ca-cert\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.086574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.086548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-envoy\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.086727 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.086709    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-podinfo\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.099100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.099067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-token\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.099357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.099336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97m2b\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-kube-api-access-97m2b\") pod \"e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") " pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.114364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.114273    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:27.141181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.141147    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:27.423347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.423313    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"]
Sep 15 19:08:27.423819 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:08:27.423724    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d4209a_caa5_43a1_a29f_5cefae17b39c.slice/crio-31f86b7168c006379b2e221a4c8d1427bd5b7a2fc4fecb1ce56420f9ea6575c4 WatchSource:0}: Error finding container 31f86b7168c006379b2e221a4c8d1427bd5b7a2fc4fecb1ce56420f9ea6575c4: Status 404 returned error can't find the container with id 31f86b7168c006379b2e221a4c8d1427bd5b7a2fc4fecb1ce56420f9ea6575c4
Sep 15 19:08:27.426873 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.426835    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:27.427000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.426905    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:27.427000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:27.426932    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:28.058052 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:28.058024    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:28.358658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:28.358539    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" event={"ID":"36d4209a-caa5-43a1-a29f-5cefae17b39c","Type":"ContainerStarted","Data":"291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"}
Sep 15 19:08:28.358837 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:28.358665    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" event={"ID":"36d4209a-caa5-43a1-a29f-5cefae17b39c","Type":"ContainerStarted","Data":"31f86b7168c006379b2e221a4c8d1427bd5b7a2fc4fecb1ce56420f9ea6575c4"}
Sep 15 19:08:28.381217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:28.381161    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podStartSLOduration=2.381144291 podStartE2EDuration="2.381144291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:28.378646168 +0000 UTC m=+1060.533014639" watchObservedRunningTime="2026-09-15 19:08:28.381144291 +0000 UTC m=+1060.535512758"
Sep 15 19:08:28.731399 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:28.731256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:29.142270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.142222    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:29.152673 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.152646    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:29.364982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.364948    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:29.366086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.366068    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:29.372676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.372651    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:29.838844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.838767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:29.938385 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:29.938328    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:30.258363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:30.258330    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:30.312931 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:30.312897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:30.640271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:30.640176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:30.750321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:30.750268    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:30.840145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:30.840109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:31.245021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.233573    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:31.245021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.242095    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:08:31.350758 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.350655    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs2k9\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-kube-api-access-rs2k9\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.350758 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.350699    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-podinfo\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.350758 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.350759    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-certs\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.350786    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-envoy\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.350802    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-data\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.350896    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-socket\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351045    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istiod-ca-cert\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351080    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-token\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351091    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:08:31.351266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351111    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-credential-socket\") pod \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\" (UID: \"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0\") "
Sep 15 19:08:31.351266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351115    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:08:31.351266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351164    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:08:31.351518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351393    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:08:31.351518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351415    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:08:31.351668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351580    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.351668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351593    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.351668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351603    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.351668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351635    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.351668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.351648    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.353252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.353224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:08:31.353252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.353241    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-kube-api-access-rs2k9" (OuterVolumeSpecName: "kube-api-access-rs2k9") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "kube-api-access-rs2k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:08:31.353419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.353300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:08:31.353532 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.353515    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" (UID: "d0d6565a-d85e-4c08-97c7-1fc61d09cdc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:08:31.381870 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.381841    2584 generic.go:356] "Generic (PLEG): container finished" podID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerID="29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe" exitCode=0
Sep 15 19:08:31.382021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.381888    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"
Sep 15 19:08:31.382021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.381929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" event={"ID":"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0","Type":"ContainerDied","Data":"29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe"}
Sep 15 19:08:31.382021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.381985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6" event={"ID":"d0d6565a-d85e-4c08-97c7-1fc61d09cdc0","Type":"ContainerDied","Data":"22d7df445ba4b4ff716f45b718aae70313ac223316ce0bdb43762b001fd5b9c0"}
Sep 15 19:08:31.382021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.382014    2584 scope.go:117] "RemoveContainer" containerID="29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe"
Sep 15 19:08:31.395365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.395346    2584 scope.go:117] "RemoveContainer" containerID="a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"
Sep 15 19:08:31.404266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.404248    2584 scope.go:117] "RemoveContainer" containerID="29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe"
Sep 15 19:08:31.404550 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:08:31.404532    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe\": container with ID starting with 29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe not found: ID does not exist" containerID="29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe"
Sep 15 19:08:31.404602 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.404558    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe"} err="failed to get container status \"29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe\": rpc error: code = NotFound desc = could not find container \"29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe\": container with ID starting with 29220ad97f857d0dd4629ad2d386413e1284b001520c5feccbe18aae1db5e1fe not found: ID does not exist"
Sep 15 19:08:31.404602 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.404574    2584 scope.go:117] "RemoveContainer" containerID="a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"
Sep 15 19:08:31.404877 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:08:31.404850    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea\": container with ID starting with a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea not found: ID does not exist" containerID="a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"
Sep 15 19:08:31.404919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.404886    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea"} err="failed to get container status \"a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea\": rpc error: code = NotFound desc = could not find container \"a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea\": container with ID starting with a8d0665753cc85f05d397e11fdc92396ee9a9947c5c7e5fd05251fb62a8b63ea not found: ID does not exist"
Sep 15 19:08:31.418723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.418693    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"]
Sep 15 19:08:31.435288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.435253    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a7a10c67-openshift-default-7b794476d-w6ln6"]
Sep 15 19:08:31.452551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.452506    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.452551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.452538    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.452551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.452550    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs2k9\" (UniqueName: \"kubernetes.io/projected/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-kube-api-access-rs2k9\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.452551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.452560    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:31.933277 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:31.933237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:32.261691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:32.254539    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" path="/var/lib/kubelet/pods/d0d6565a-d85e-4c08-97c7-1fc61d09cdc0/volumes"
Sep 15 19:08:32.815827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:32.815790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:33.729271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:33.729238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:34.549134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.549101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:34.606380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.606349    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"]
Sep 15 19:08:34.606822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.606809    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy"
Sep 15 19:08:34.606883 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.606824    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy"
Sep 15 19:08:34.606919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.606904    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy"
Sep 15 19:08:34.606919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.606914    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy"
Sep 15 19:08:34.610314 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.610292    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.613107 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.613084    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"kube-root-ca.crt\""
Sep 15 19:08:34.613232 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.613142    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"openshift-service-ca.crt\""
Sep 15 19:08:34.614496 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.614450    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"simulator-w0-900c88-kserve-self-signed-certs\""
Sep 15 19:08:34.614590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.614475    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"default-dockercfg-6plq7\""
Sep 15 19:08:34.620253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.620183    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"]
Sep 15 19:08:34.792435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.792399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-home\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.792435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.792441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-dshm\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.792899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.792463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdqs\" (UniqueName: \"kubernetes.io/projected/999e7660-0f32-489b-9d36-1d84211bfd55-kube-api-access-fmdqs\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.792899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.792580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/999e7660-0f32-489b-9d36-1d84211bfd55-tls-certs\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.792899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.792713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-model-cache\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.792899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.792870    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-tmp-dir\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.870047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.869969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:34.893650 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.893592    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-model-cache\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.893822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.893752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-tmp-dir\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.893822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.893791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-home\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.893822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.893819    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-dshm\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.893994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.893850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdqs\" (UniqueName: \"kubernetes.io/projected/999e7660-0f32-489b-9d36-1d84211bfd55-kube-api-access-fmdqs\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.893994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.893903    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/999e7660-0f32-489b-9d36-1d84211bfd55-tls-certs\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.894113 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.894012    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-model-cache\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.894185 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.894164    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-tmp-dir\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.894273 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.894247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-home\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.896201 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.896168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-dshm\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.896371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.896352    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/999e7660-0f32-489b-9d36-1d84211bfd55-tls-certs\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.901175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.901148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdqs\" (UniqueName: \"kubernetes.io/projected/999e7660-0f32-489b-9d36-1d84211bfd55-kube-api-access-fmdqs\") pod \"simulator-w0-900c88-kserve-57ffb4869c-95w4d\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") " pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.922939 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.922900    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:34.983103 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:34.983064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:35.218817 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:35.218796    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"]
Sep 15 19:08:35.246096 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:35.246038    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:35.414921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:35.414884    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d" event={"ID":"999e7660-0f32-489b-9d36-1d84211bfd55","Type":"ContainerStarted","Data":"e4df411d88b3f2134126dbb0acff6ce1150964c091b778003759769d6ef0decc"}
Sep 15 19:08:35.415072 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:35.414931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d" event={"ID":"999e7660-0f32-489b-9d36-1d84211bfd55","Type":"ContainerStarted","Data":"ac9ca0c712a077478e971713a69969252bb8bac3eba0dd7be7aebda445ca724b"}
Sep 15 19:08:35.433715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:35.433661    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d" podStartSLOduration=1.433640214 podStartE2EDuration="1.433640214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:08:35.43171173 +0000 UTC m=+1067.586080199" watchObservedRunningTime="2026-09-15 19:08:35.433640214 +0000 UTC m=+1067.588008685"
Sep 15 19:08:36.423948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:36.423909    2584 generic.go:356] "Generic (PLEG): container finished" podID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerID="701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133" exitCode=0
Sep 15 19:08:36.424457 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:36.423993    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" event={"ID":"983248b6-b680-4c9e-bb1e-bc22ae769201","Type":"ContainerDied","Data":"701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"}
Sep 15 19:08:36.424596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:36.424577    2584 scope.go:117] "RemoveContainer" containerID="701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"
Sep 15 19:08:36.425868 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:36.425840    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:36.425954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:36.425906    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:36.425954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:36.425937    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:37.437849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:37.437804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" event={"ID":"983248b6-b680-4c9e-bb1e-bc22ae769201","Type":"ContainerStarted","Data":"0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f"}
Sep 15 19:08:37.703284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:37.703187    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:37.703486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:37.703455    2584 patch_prober.go:28] interesting pod/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 15 19:08:37.703571 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:37.703525    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 15 19:08:38.703889 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:38.703852    2584 patch_prober.go:28] interesting pod/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 15 19:08:38.704274 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:38.703909    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 15 19:08:39.449325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:39.449281    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:39.704063 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:39.703970    2584 patch_prober.go:28] interesting pod/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 15 19:08:39.704063 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:39.704036    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 15 19:08:40.704223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:40.704190    2584 patch_prober.go:28] interesting pod/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 15 19:08:40.704636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:40.704275    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 15 19:08:41.704129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:41.704091    2584 patch_prober.go:28] interesting pod/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 15 19:08:41.704319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:41.704160    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 15 19:08:42.708814 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:42.708780    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:42.709199 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:42.709161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:42.709938 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:42.709917    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:08:42.776025 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:42.775980    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:42.874395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:42.874365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:44.923830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:44.923787    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:44.924334 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:44.923854    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:44.957996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:44.957965    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:45.507770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:45.507740    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:08:51.101798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:51.101757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:52.309485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.309445    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:52.545537 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.545501    2584 generic.go:356] "Generic (PLEG): container finished" podID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerID="291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7" exitCode=0
Sep 15 19:08:52.545749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.545575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" event={"ID":"36d4209a-caa5-43a1-a29f-5cefae17b39c","Type":"ContainerDied","Data":"291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"}
Sep 15 19:08:52.546073 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.546059    2584 scope.go:117] "RemoveContainer" containerID="291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"
Sep 15 19:08:52.547271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.547242    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:52.547333 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.547312    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:52.547365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.547347    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:08:52.930695 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:52.930593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:53.554169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:53.554133    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" event={"ID":"36d4209a-caa5-43a1-a29f-5cefae17b39c","Type":"ContainerStarted","Data":"d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9"}
Sep 15 19:08:53.563495 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:53.563466    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:54.141721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.141682    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:08:54.142159 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.142124    2584 patch_prober.go:28] interesting pod/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 19:08:54.142275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.142189    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 19:08:54.298068 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.298011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:54.755210 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.755171    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"]
Sep 15 19:08:54.755654 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.755639    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy"
Sep 15 19:08:54.755654 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.755655    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d6565a-d85e-4c08-97c7-1fc61d09cdc0" containerName="istio-proxy"
Sep 15 19:08:54.757812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.757796    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:54.760122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.760102    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:08:54.760215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.760127    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:08:54.767011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.766984    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"]
Sep 15 19:08:54.936887 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:54.936839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4275555-8692-41b8-a995-1113a2dcf995-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq\" (UID: \"a4275555-8692-41b8-a995-1113a2dcf995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:55.038657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.038535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4275555-8692-41b8-a995-1113a2dcf995-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq\" (UID: \"a4275555-8692-41b8-a995-1113a2dcf995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:55.039481 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.039448    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4275555-8692-41b8-a995-1113a2dcf995-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq\" (UID: \"a4275555-8692-41b8-a995-1113a2dcf995\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:55.070574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.070542    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:55.159985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.159292    2584 patch_prober.go:28] interesting pod/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 19:08:55.159985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.159404    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 19:08:55.236357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.236213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:55.363992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.363850    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"]
Sep 15 19:08:55.367497 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:08:55.367435    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4275555_8692_41b8_a995_1113a2dcf995.slice/crio-5257dce396443a260f73c3c0093641f1a5388f8b4a41af457db5382fbc27618f WatchSource:0}: Error finding container 5257dce396443a260f73c3c0093641f1a5388f8b4a41af457db5382fbc27618f: Status 404 returned error can't find the container with id 5257dce396443a260f73c3c0093641f1a5388f8b4a41af457db5382fbc27618f
Sep 15 19:08:55.569471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.569426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq" event={"ID":"a4275555-8692-41b8-a995-1113a2dcf995","Type":"ContainerStarted","Data":"5257dce396443a260f73c3c0093641f1a5388f8b4a41af457db5382fbc27618f"}
Sep 15 19:08:55.881895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:55.881851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:56.142145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:56.141646    2584 patch_prober.go:28] interesting pod/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 19:08:56.142145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:56.141733    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 19:08:56.577947 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:56.577907    2584 generic.go:356] "Generic (PLEG): container finished" podID="a4275555-8692-41b8-a995-1113a2dcf995" containerID="884365e120d82916d332ef6f04223123d76af62a275d78682548a2c0a7312423" exitCode=0
Sep 15 19:08:56.578129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:56.578068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq" event={"ID":"a4275555-8692-41b8-a995-1113a2dcf995","Type":"ContainerDied","Data":"884365e120d82916d332ef6f04223123d76af62a275d78682548a2c0a7312423"}
Sep 15 19:08:56.639356 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:56.639314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:57.141642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.141582    2584 patch_prober.go:28] interesting pod/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 19:08:57.142298 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.142262    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 19:08:57.345272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.345236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:57.778533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.778498    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:57.879206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.879120    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4275555-8692-41b8-a995-1113a2dcf995-maas-api-service-ca\") pod \"a4275555-8692-41b8-a995-1113a2dcf995\" (UID: \"a4275555-8692-41b8-a995-1113a2dcf995\") "
Sep 15 19:08:57.879576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.879554    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4275555-8692-41b8-a995-1113a2dcf995-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4275555-8692-41b8-a995-1113a2dcf995" (UID: "a4275555-8692-41b8-a995-1113a2dcf995"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:08:57.980461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.980427    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4275555-8692-41b8-a995-1113a2dcf995-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:08:57.999493 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:57.999463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:58.142522 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:58.142408    2584 patch_prober.go:28] interesting pod/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 19:08:58.142522 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:58.142481    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 19:08:58.592133 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:58.592098    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq" event={"ID":"a4275555-8692-41b8-a995-1113a2dcf995","Type":"ContainerDied","Data":"5257dce396443a260f73c3c0093641f1a5388f8b4a41af457db5382fbc27618f"}
Sep 15 19:08:58.592348 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:58.592154    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5257dce396443a260f73c3c0093641f1a5388f8b4a41af457db5382fbc27618f"
Sep 15 19:08:58.592348 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:58.592222    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"
Sep 15 19:08:58.757444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:58.757328    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:08:59.141921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:59.141875    2584 patch_prober.go:28] interesting pod/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 15 19:08:59.142138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:59.141946    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 15 19:08:59.365579 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:08:59.365539    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:09:00.146382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:00.146352    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:09:00.147642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:00.147601    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:09:01.136376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.136330    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:01.667586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.667548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:01.768066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.768001    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:01.846516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.846479    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"]
Sep 15 19:09:01.847341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.847309    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:01.849245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.849224    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8b287f5b-b9jsq"]
Sep 15 19:09:01.911857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.911826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:01.993282 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.993197    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"]
Sep 15 19:09:01.993601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:01.993570    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy" containerID="cri-o://d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9" gracePeriod=30
Sep 15 19:09:02.013596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.013556    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:02.235922 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.235891    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4275555-8692-41b8-a995-1113a2dcf995" path="/var/lib/kubelet/pods/a4275555-8692-41b8-a995-1113a2dcf995/volumes"
Sep 15 19:09:02.305888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.305855    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:02.428101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.428066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:02.703840 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.703802    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"]
Sep 15 19:09:02.704448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.704429    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4275555-8692-41b8-a995-1113a2dcf995" containerName="revoke-keys"
Sep 15 19:09:02.704448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.704447    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4275555-8692-41b8-a995-1113a2dcf995" containerName="revoke-keys"
Sep 15 19:09:02.704594 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.704570    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4275555-8692-41b8-a995-1113a2dcf995" containerName="revoke-keys"
Sep 15 19:09:02.707809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.707786    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.710286 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.710264    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-140d8435-openshift-default-dockercfg-56wpl\""
Sep 15 19:09:02.721946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.721892    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"]
Sep 15 19:09:02.733952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.733923    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-podinfo\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734125 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.733964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8vn\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-kube-api-access-lq8vn\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734125 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.733993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee06b9b-0aed-47d8-a431-7c16708896db-istiod-ca-cert\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734230 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.734137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-socket\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.734273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-envoy\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.734330    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-token\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734408 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.734389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-certs\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.734442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-credential-socket\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.734514 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.734474    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-data\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-podinfo\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8vn\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-kube-api-access-lq8vn\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee06b9b-0aed-47d8-a431-7c16708896db-istiod-ca-cert\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-socket\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-envoy\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835564    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-token\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-certs\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-credential-socket\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.835809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.835723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-data\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.836152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.836115    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-data\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.836484 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.836457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-certs\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.836712 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.836688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-credential-socket\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.836851 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.836757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-socket\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.840632 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.837210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee06b9b-0aed-47d8-a431-7c16708896db-istiod-ca-cert\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.840632 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.837257    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:02.842754 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.842703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-podinfo\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.846369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.846344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8vn\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-kube-api-access-lq8vn\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.848470 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.848442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-token\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:02.848839 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:02.848817    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-envoy\") pod \"e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") " pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:03.011380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.011340    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:03.022208 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.022181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:03.316229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.316204    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"]
Sep 15 19:09:03.318128 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:09:03.318081    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee06b9b_0aed_47d8_a431_7c16708896db.slice/crio-b827e3ecc5ead22a7fcb59c9772b61a93b9c8e532f5a8cc525f8b1a352c119b7 WatchSource:0}: Error finding container b827e3ecc5ead22a7fcb59c9772b61a93b9c8e532f5a8cc525f8b1a352c119b7: Status 404 returned error can't find the container with id b827e3ecc5ead22a7fcb59c9772b61a93b9c8e532f5a8cc525f8b1a352c119b7
Sep 15 19:09:03.325001 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.324952    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:03.325138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.325061    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:03.325138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.325103    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:03.573196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.573158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:03.629821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.629676    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" event={"ID":"4ee06b9b-0aed-47d8-a431-7c16708896db","Type":"ContainerStarted","Data":"72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"}
Sep 15 19:09:03.629821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.629763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" event={"ID":"4ee06b9b-0aed-47d8-a431-7c16708896db","Type":"ContainerStarted","Data":"b827e3ecc5ead22a7fcb59c9772b61a93b9c8e532f5a8cc525f8b1a352c119b7"}
Sep 15 19:09:03.651519 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.651419    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podStartSLOduration=1.651397 podStartE2EDuration="1.651397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:03.651159328 +0000 UTC m=+1095.805527793" watchObservedRunningTime="2026-09-15 19:09:03.651397 +0000 UTC m=+1095.805765469"
Sep 15 19:09:03.655541 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:03.655508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:04.023170 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:04.023133    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:04.030073 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:04.030043    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:04.490600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:04.490508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:04.638079 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:04.638052    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:04.639161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:04.639139    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:05.012695 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:05.012652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:05.110479 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:05.110446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:05.323522 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:05.323424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:06.013397 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:06.013365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:06.711827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:06.711795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:07.349572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.349546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:07.351433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.351417    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:09:07.390067 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390032    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97m2b\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-kube-api-access-97m2b\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-envoy\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390137    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-podinfo\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390174    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-socket\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390228    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4209a-caa5-43a1-a29f-5cefae17b39c-istiod-ca-cert\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390279    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-certs\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390314    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-data\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390339    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-token\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-credential-socket\") pod \"36d4209a-caa5-43a1-a29f-5cefae17b39c\" (UID: \"36d4209a-caa5-43a1-a29f-5cefae17b39c\") "
Sep 15 19:09:07.390491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390459    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:07.390766 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390717    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-data" (OuterVolumeSpecName: "istio-data") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:07.390831 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390748    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d4209a-caa5-43a1-a29f-5cefae17b39c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:07.391191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390929    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.391191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390952    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d4209a-caa5-43a1-a29f-5cefae17b39c-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.391191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390956    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:07.391191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390966    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:07.391191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.390966    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.392907 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.392883    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-kube-api-access-97m2b" (OuterVolumeSpecName: "kube-api-access-97m2b") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "kube-api-access-97m2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:07.393052 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.392881    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:07.393340 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.393319    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-token" (OuterVolumeSpecName: "istio-token") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:07.393476 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.393461    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36d4209a-caa5-43a1-a29f-5cefae17b39c" (UID: "36d4209a-caa5-43a1-a29f-5cefae17b39c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:09:07.492666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.492547    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.492666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.492576    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.492666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.492585    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.492666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.492596    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97m2b\" (UniqueName: \"kubernetes.io/projected/36d4209a-caa5-43a1-a29f-5cefae17b39c-kube-api-access-97m2b\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.492666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.492607    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.492666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.492630    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d4209a-caa5-43a1-a29f-5cefae17b39c-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:07.660944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.660906    2584 generic.go:356] "Generic (PLEG): container finished" podID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerID="d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9" exitCode=0
Sep 15 19:09:07.661089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.660955    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"
Sep 15 19:09:07.661089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.660987    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" event={"ID":"36d4209a-caa5-43a1-a29f-5cefae17b39c","Type":"ContainerDied","Data":"d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9"}
Sep 15 19:09:07.661089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.661046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s" event={"ID":"36d4209a-caa5-43a1-a29f-5cefae17b39c","Type":"ContainerDied","Data":"31f86b7168c006379b2e221a4c8d1427bd5b7a2fc4fecb1ce56420f9ea6575c4"}
Sep 15 19:09:07.661089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.661072    2584 scope.go:117] "RemoveContainer" containerID="d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9"
Sep 15 19:09:07.671581 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.671557    2584 scope.go:117] "RemoveContainer" containerID="291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"
Sep 15 19:09:07.680676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.680650    2584 scope.go:117] "RemoveContainer" containerID="d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9"
Sep 15 19:09:07.680948 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:09:07.680928    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9\": container with ID starting with d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9 not found: ID does not exist" containerID="d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9"
Sep 15 19:09:07.681037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.680959    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9"} err="failed to get container status \"d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9\": rpc error: code = NotFound desc = could not find container \"d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9\": container with ID starting with d1c34b5369662af0b95400830d0cecb3b49f3f3e254dfb6a956573d018ccc3b9 not found: ID does not exist"
Sep 15 19:09:07.681037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.680983    2584 scope.go:117] "RemoveContainer" containerID="291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"
Sep 15 19:09:07.681244 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:09:07.681221    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7\": container with ID starting with 291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7 not found: ID does not exist" containerID="291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"
Sep 15 19:09:07.681288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.681255    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7"} err="failed to get container status \"291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7\": rpc error: code = NotFound desc = could not find container \"291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7\": container with ID starting with 291d3ac505dac3e06cde580b61e4850f253becdcfa557914fa94cd74e4159db7 not found: ID does not exist"
Sep 15 19:09:07.683090 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.683071    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"]
Sep 15 19:09:07.688930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:07.688906    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8b287f5b-gw-openshift-default-76c69bbb4d-pcs8s"]
Sep 15 19:09:08.225034 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:08.224998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:08.236288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:08.236255    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" path="/var/lib/kubelet/pods/36d4209a-caa5-43a1-a29f-5cefae17b39c/volumes"
Sep 15 19:09:08.281894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:08.281859    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:08.425082 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:08.425047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:08.772887 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:08.772849    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:08.868677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:08.868644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:27.811439 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:27.811400    2584 generic.go:356] "Generic (PLEG): container finished" podID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerID="72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337" exitCode=0
Sep 15 19:09:27.811898 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:27.811469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" event={"ID":"4ee06b9b-0aed-47d8-a431-7c16708896db","Type":"ContainerDied","Data":"72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"}
Sep 15 19:09:27.811943 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:27.811908    2584 scope.go:117] "RemoveContainer" containerID="72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"
Sep 15 19:09:27.813101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:27.813073    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:27.813163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:27.813143    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:27.813197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:27.813176    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:28.821235 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:28.821169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" event={"ID":"4ee06b9b-0aed-47d8-a431-7c16708896db","Type":"ContainerStarted","Data":"990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8"}
Sep 15 19:09:29.024032 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:29.023992    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:29.024328 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:29.024292    2584 patch_prober.go:28] interesting pod/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 19:09:29.024565 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:29.024539    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 19:09:30.023677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.023634    2584 patch_prober.go:28] interesting pod/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 19:09:30.024201 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.023724    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 19:09:30.221109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.221072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:30.675317 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.675283    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"]
Sep 15 19:09:30.675762 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.675748    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy"
Sep 15 19:09:30.675818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.675764    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy"
Sep 15 19:09:30.675818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.675774    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy"
Sep 15 19:09:30.675818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.675780    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy"
Sep 15 19:09:30.675918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.675853    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy"
Sep 15 19:09:30.678205 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.678189    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:30.680443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.680419    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:09:30.680577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.680421    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:09:30.685688 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.685656    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"]
Sep 15 19:09:30.750467 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.750423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e3e23d-89cd-494e-85a5-33ce3048ee18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-140d8435-6g86h\" (UID: \"22e3e23d-89cd-494e-85a5-33ce3048ee18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:30.852679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.852607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e3e23d-89cd-494e-85a5-33ce3048ee18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-140d8435-6g86h\" (UID: \"22e3e23d-89cd-494e-85a5-33ce3048ee18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:30.853298 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.853270    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e3e23d-89cd-494e-85a5-33ce3048ee18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-140d8435-6g86h\" (UID: \"22e3e23d-89cd-494e-85a5-33ce3048ee18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:30.950200 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.950109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:30.991091 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:30.991053    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:31.026490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.026012    2584 patch_prober.go:28] interesting pod/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 19:09:31.026490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.026080    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 19:09:31.296582 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.296551    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"]
Sep 15 19:09:31.297706 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:09:31.297678    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e3e23d_89cd_494e_85a5_33ce3048ee18.slice/crio-d590719af5ebcae1b9a7f0454f1959ad8a61f47e855d3c03095a4be81fc55e01 WatchSource:0}: Error finding container d590719af5ebcae1b9a7f0454f1959ad8a61f47e855d3c03095a4be81fc55e01: Status 404 returned error can't find the container with id d590719af5ebcae1b9a7f0454f1959ad8a61f47e855d3c03095a4be81fc55e01
Sep 15 19:09:31.715996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.715917    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:31.854243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.854197    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h" event={"ID":"22e3e23d-89cd-494e-85a5-33ce3048ee18","Type":"ContainerStarted","Data":"bdf6305a96d281110071f2abf26709c8611fc9a49b4349ae61cb648bef2e054a"}
Sep 15 19:09:31.854557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.854517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h" event={"ID":"22e3e23d-89cd-494e-85a5-33ce3048ee18","Type":"ContainerStarted","Data":"d590719af5ebcae1b9a7f0454f1959ad8a61f47e855d3c03095a4be81fc55e01"}
Sep 15 19:09:31.871357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:31.871307    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h" podStartSLOduration=1.8712931350000002 podStartE2EDuration="1.871293135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:31.86871338 +0000 UTC m=+1124.023081849" watchObservedRunningTime="2026-09-15 19:09:31.871293135 +0000 UTC m=+1124.025661646"
Sep 15 19:09:32.023148 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:32.023107    2584 patch_prober.go:28] interesting pod/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 19:09:32.023357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:32.023181    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 19:09:32.452334 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:32.452302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:32.566181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:32.566149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:32.862982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:32.862949    2584 generic.go:356] "Generic (PLEG): container finished" podID="22e3e23d-89cd-494e-85a5-33ce3048ee18" containerID="bdf6305a96d281110071f2abf26709c8611fc9a49b4349ae61cb648bef2e054a" exitCode=0
Sep 15 19:09:32.862982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:32.862989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h" event={"ID":"22e3e23d-89cd-494e-85a5-33ce3048ee18","Type":"ContainerDied","Data":"bdf6305a96d281110071f2abf26709c8611fc9a49b4349ae61cb648bef2e054a"}
Sep 15 19:09:33.023784 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:33.023745    2584 patch_prober.go:28] interesting pod/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 19:09:33.024073 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:33.024043    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 19:09:33.240198 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:33.240104    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:34.034689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.034651    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:34.035138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.034976    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:34.061417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.061272    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:34.061822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.061787    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:34.062648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.062630    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:34.084714 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.084691    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e3e23d-89cd-494e-85a5-33ce3048ee18-maas-api-service-ca\") pod \"22e3e23d-89cd-494e-85a5-33ce3048ee18\" (UID: \"22e3e23d-89cd-494e-85a5-33ce3048ee18\") "
Sep 15 19:09:34.085083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.085056    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e3e23d-89cd-494e-85a5-33ce3048ee18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22e3e23d-89cd-494e-85a5-33ce3048ee18" (UID: "22e3e23d-89cd-494e-85a5-33ce3048ee18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:34.186047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.185966    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e3e23d-89cd-494e-85a5-33ce3048ee18-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:34.815529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.815496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:34.882413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.882373    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"
Sep 15 19:09:34.883129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.883095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h" event={"ID":"22e3e23d-89cd-494e-85a5-33ce3048ee18","Type":"ContainerDied","Data":"d590719af5ebcae1b9a7f0454f1959ad8a61f47e855d3c03095a4be81fc55e01"}
Sep 15 19:09:34.883284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:34.883168    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d590719af5ebcae1b9a7f0454f1959ad8a61f47e855d3c03095a4be81fc55e01"
Sep 15 19:09:35.547050 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:35.547016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:36.353306 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:36.353266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:37.051760 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.051726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:37.153357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.153327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:37.236241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.236209    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"]
Sep 15 19:09:37.238045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.238015    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:37.239145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.239124    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-140d8435-6g86h"]
Sep 15 19:09:37.357017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.356934    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:37.644175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:37.644087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:38.234494 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:38.234463    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e3e23d-89cd-494e-85a5-33ce3048ee18" path="/var/lib/kubelet/pods/22e3e23d-89cd-494e-85a5-33ce3048ee18/volumes"
Sep 15 19:09:38.338232 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:38.338199    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"]
Sep 15 19:09:38.338479 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:38.338451    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy" containerID="cri-o://990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8" gracePeriod=30
Sep 15 19:09:38.339566 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:38.339540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:38.492269 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:38.492184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:38.785412 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:38.785381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:39.052708 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.052590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:39.078270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.078234    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"]
Sep 15 19:09:39.078927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.078910    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e3e23d-89cd-494e-85a5-33ce3048ee18" containerName="revoke-keys"
Sep 15 19:09:39.079009 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.078930    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e3e23d-89cd-494e-85a5-33ce3048ee18" containerName="revoke-keys"
Sep 15 19:09:39.079077 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.079066    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e3e23d-89cd-494e-85a5-33ce3048ee18" containerName="revoke-keys"
Sep 15 19:09:39.079115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.079080    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d4209a-caa5-43a1-a29f-5cefae17b39c" containerName="istio-proxy"
Sep 15 19:09:39.081793 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.081774    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.084108 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.084088    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-282a6587-openshift-default-dockercfg-fv6sb\""
Sep 15 19:09:39.092162 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.092137    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"]
Sep 15 19:09:39.145828 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.145795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:39.155129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-podinfo\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wkb\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-kube-api-access-r8wkb\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-socket\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-credential-socket\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-certs\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37e673-7f67-4d1e-af71-d1eb8abac355-istiod-ca-cert\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155313    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-token\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-data\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.155567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.155427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-envoy\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.256683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-envoy\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-podinfo\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wkb\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-kube-api-access-r8wkb\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-socket\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-credential-socket\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-certs\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37e673-7f67-4d1e-af71-d1eb8abac355-istiod-ca-cert\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-token\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.256949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-data\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.257209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-socket\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.257354    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-credential-socket\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.257469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-certs\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.257548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-data\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.257888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.257710    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37e673-7f67-4d1e-af71-d1eb8abac355-istiod-ca-cert\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.259264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.259236    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-envoy\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.259406 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.259392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-podinfo\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.267590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.267565    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-token\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.268056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.268033    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wkb\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-kube-api-access-r8wkb\") pod \"e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") " pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.394092 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.394006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:39.679651 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.678336    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"]
Sep 15 19:09:39.684907 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.684875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:39.686156 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.686118    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:39.686380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.686365    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:39.686430 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.686402    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:09:39.881740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.881704    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:39.917954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.917918    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" event={"ID":"dd37e673-7f67-4d1e-af71-d1eb8abac355","Type":"ContainerStarted","Data":"ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"}
Sep 15 19:09:39.918115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.917975    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" event={"ID":"dd37e673-7f67-4d1e-af71-d1eb8abac355","Type":"ContainerStarted","Data":"56ca71c4989802a81c1fd382214bdf56f53eaa8316332e4f69d4c4be9f25c8d1"}
Sep 15 19:09:39.940454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:39.940343    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podStartSLOduration=0.940326445 podStartE2EDuration="940.326445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:39.93645199 +0000 UTC m=+1132.090820458" watchObservedRunningTime="2026-09-15 19:09:39.940326445 +0000 UTC m=+1132.094694913"
Sep 15 19:09:40.394767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:40.394729    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:40.400355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:40.400329    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:40.672119 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:40.672039    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:40.926161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:40.926067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:40.927208 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:40.927177    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:09:41.434871 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:41.434831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:41.537149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:41.537102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:42.054164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:42.054133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:43.126932 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.126903    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:43.604584 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.604552    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:43.702434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702328    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-socket\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702387    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-certs\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702424    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq8vn\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-kube-api-access-lq8vn\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702492    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-data\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702532    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-envoy\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702642    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:43.702752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702681    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-token\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702727    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-credential-socket\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.702752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702732    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:43.703059 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702788    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-podinfo\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.703059 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702813    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee06b9b-0aed-47d8-a431-7c16708896db-istiod-ca-cert\") pod \"4ee06b9b-0aed-47d8-a431-7c16708896db\" (UID: \"4ee06b9b-0aed-47d8-a431-7c16708896db\") "
Sep 15 19:09:43.703059 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702824    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:43.703059 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.702995    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:43.703221 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.703206    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee06b9b-0aed-47d8-a431-7c16708896db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:09:43.703369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.703340    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.703369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.703365    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.703469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.703382    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee06b9b-0aed-47d8-a431-7c16708896db-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.703469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.703397    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.703469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.703410    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.704890 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.704865    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:09:43.704990 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.704939    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:09:43.705179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.705158    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-kube-api-access-lq8vn" (OuterVolumeSpecName: "kube-api-access-lq8vn") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "kube-api-access-lq8vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:43.705272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.705185    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ee06b9b-0aed-47d8-a431-7c16708896db" (UID: "4ee06b9b-0aed-47d8-a431-7c16708896db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:09:43.804424 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.804388    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8vn\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-kube-api-access-lq8vn\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.804424 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.804419    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.804424 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.804429    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.804688 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.804437    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee06b9b-0aed-47d8-a431-7c16708896db-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:09:43.831123 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.831088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:43.947850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.947817    2584 generic.go:356] "Generic (PLEG): container finished" podID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerID="990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8" exitCode=0
Sep 15 19:09:43.948020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.947873    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"
Sep 15 19:09:43.948020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.947895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" event={"ID":"4ee06b9b-0aed-47d8-a431-7c16708896db","Type":"ContainerDied","Data":"990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8"}
Sep 15 19:09:43.948020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.947948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n" event={"ID":"4ee06b9b-0aed-47d8-a431-7c16708896db","Type":"ContainerDied","Data":"b827e3ecc5ead22a7fcb59c9772b61a93b9c8e532f5a8cc525f8b1a352c119b7"}
Sep 15 19:09:43.948020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.947972    2584 scope.go:117] "RemoveContainer" containerID="990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8"
Sep 15 19:09:43.958334 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.958314    2584 scope.go:117] "RemoveContainer" containerID="72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"
Sep 15 19:09:43.968226 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.968207    2584 scope.go:117] "RemoveContainer" containerID="990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8"
Sep 15 19:09:43.968547 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:09:43.968522    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8\": container with ID starting with 990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8 not found: ID does not exist" containerID="990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8"
Sep 15 19:09:43.968730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.968559    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8"} err="failed to get container status \"990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8\": rpc error: code = NotFound desc = could not find container \"990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8\": container with ID starting with 990e93d2b5db28650f3649bea6d22854a11c4789babd6acb7885859565f594f8 not found: ID does not exist"
Sep 15 19:09:43.968730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.968585    2584 scope.go:117] "RemoveContainer" containerID="72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"
Sep 15 19:09:43.968946 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:09:43.968927    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337\": container with ID starting with 72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337 not found: ID does not exist" containerID="72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"
Sep 15 19:09:43.968984 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.968954    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337"} err="failed to get container status \"72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337\": rpc error: code = NotFound desc = could not find container \"72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337\": container with ID starting with 72d2d40c0dca6fb1873f65a9a19e5f3bad1970d74cef274f639013c962892337 not found: ID does not exist"
Sep 15 19:09:43.970877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.970855    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"]
Sep 15 19:09:43.979591 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:43.979562    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-140d8435-openshift-default-747f6bdc8b-p6w6n"]
Sep 15 19:09:44.234564 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:44.234480    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" path="/var/lib/kubelet/pods/4ee06b9b-0aed-47d8-a431-7c16708896db/volumes"
Sep 15 19:09:44.627876 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:44.627838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:44.727965 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:44.727933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:44.845197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:44.845161    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:45.104414 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:45.104376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:45.198726 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:45.198688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:49.866806 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.866768    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"]
Sep 15 19:09:49.867275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.867256    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy"
Sep 15 19:09:49.867325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.867275    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy"
Sep 15 19:09:49.867325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.867289    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy"
Sep 15 19:09:49.867325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.867295    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy"
Sep 15 19:09:49.867419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.867371    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy"
Sep 15 19:09:49.867419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.867386    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee06b9b-0aed-47d8-a431-7c16708896db" containerName="istio-proxy"
Sep 15 19:09:49.870244 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.870228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:49.873658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.873635    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9de44d\"/\"default-dockercfg-bmgvs\""
Sep 15 19:09:49.873658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.873659    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9de44d\"/\"openshift-service-ca.crt\""
Sep 15 19:09:49.873849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.873678    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9de44d\"/\"test-model-9de44d-kserve-self-signed-certs\""
Sep 15 19:09:49.873849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.873692    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9de44d\"/\"kube-root-ca.crt\""
Sep 15 19:09:49.880098 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.880069    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"]
Sep 15 19:09:49.971705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.971661    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tmp-dir\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:49.971872 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.971726    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nsfv\" (UniqueName: \"kubernetes.io/projected/d5fa3580-f5f8-4ecc-97f7-b491ca162576-kube-api-access-7nsfv\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:49.971872 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.971753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-model-cache\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:49.971872 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.971770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tls-certs\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:49.972007 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.971911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-home\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:49.972007 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:49.971943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-dshm\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tmp-dir\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nsfv\" (UniqueName: \"kubernetes.io/projected/d5fa3580-f5f8-4ecc-97f7-b491ca162576-kube-api-access-7nsfv\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-model-cache\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tls-certs\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-home\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073526 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073345    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-dshm\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073583 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tmp-dir\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073583 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073560    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-home\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.073713 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.073642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-model-cache\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.075512 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.075484    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-dshm\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.075835 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.075819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tls-certs\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.092289 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.092263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nsfv\" (UniqueName: \"kubernetes.io/projected/d5fa3580-f5f8-4ecc-97f7-b491ca162576-kube-api-access-7nsfv\") pod \"test-model-9de44d-kserve-7789d66fc-qsq55\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") " pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.181092 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.180991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:09:50.367083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.367049    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:09:50.458923 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.458894    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"]
Sep 15 19:09:50.459824 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:09:50.459798    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5fa3580_f5f8_4ecc_97f7_b491ca162576.slice/crio-fe4b2d10a23e580de6a8d57de7ba83626a3560fa0b6893ca6e1540a2900f2a88 WatchSource:0}: Error finding container fe4b2d10a23e580de6a8d57de7ba83626a3560fa0b6893ca6e1540a2900f2a88: Status 404 returned error can't find the container with id fe4b2d10a23e580de6a8d57de7ba83626a3560fa0b6893ca6e1540a2900f2a88
Sep 15 19:09:50.996190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.996154    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55" event={"ID":"d5fa3580-f5f8-4ecc-97f7-b491ca162576","Type":"ContainerStarted","Data":"aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876"}
Sep 15 19:09:50.996190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:50.996199    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55" event={"ID":"d5fa3580-f5f8-4ecc-97f7-b491ca162576","Type":"ContainerStarted","Data":"fe4b2d10a23e580de6a8d57de7ba83626a3560fa0b6893ca6e1540a2900f2a88"}
Sep 15 19:09:51.014957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:09:51.014901    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55" podStartSLOduration=2.01488668 podStartE2EDuration="2.01488668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:09:51.011437209 +0000 UTC m=+1143.165805678" watchObservedRunningTime="2026-09-15 19:09:51.01488668 +0000 UTC m=+1143.169255147"
Sep 15 19:10:00.181905 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:00.181865    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:10:00.181905 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:00.181917    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:10:00.194802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:00.194775    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:10:01.087298 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:01.087266    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:10:04.097554 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:04.097521    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerID="ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181" exitCode=0
Sep 15 19:10:04.098105 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:04.097599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" event={"ID":"dd37e673-7f67-4d1e-af71-d1eb8abac355","Type":"ContainerDied","Data":"ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"}
Sep 15 19:10:04.098195 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:04.098177    2584 scope.go:117] "RemoveContainer" containerID="ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"
Sep 15 19:10:04.099420 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:04.099388    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:10:04.099506 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:04.099462    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:10:04.099506 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:04.099492    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:10:05.105682 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:05.105649    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" event={"ID":"dd37e673-7f67-4d1e-af71-d1eb8abac355","Type":"ContainerStarted","Data":"121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55"}
Sep 15 19:10:05.394839 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:05.394743    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:10:05.395011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:05.394987    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 19:10:05.395071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:05.395050    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 19:10:06.394826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:06.394792    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 19:10:06.395196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:06.394858    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 19:10:06.547008 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:06.546972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:07.110050 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:07.110009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:07.394828 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:07.394727    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 19:10:07.394828 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:07.394806    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 19:10:07.815833 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:07.815796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:08.395444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:08.395400    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 19:10:08.396022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:08.395476    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 19:10:08.597173 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:08.597141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:09.107444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:09.107412    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:09.394996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:09.394905    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 19:10:09.394996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:09.394970    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 19:10:09.730955 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:09.730866    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:10.395358 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:10.395312    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 15 19:10:10.395549 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:10.395372    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 15 19:10:10.926578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:10.926487    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:10:11.183430 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.183352    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"]
Sep 15 19:10:11.186107 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.186090    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:11.188184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.188162    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:10:11.188184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.188180    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:10:11.194164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.194135    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"]
Sep 15 19:10:11.306020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.305982    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d890a150-fb5d-466d-ba71-9c08d69f7efe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-282a6587-vlltv\" (UID: \"d890a150-fb5d-466d-ba71-9c08d69f7efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:11.398933 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.398901    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:10:11.399832 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.399814    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:10:11.407080 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.407056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d890a150-fb5d-466d-ba71-9c08d69f7efe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-282a6587-vlltv\" (UID: \"d890a150-fb5d-466d-ba71-9c08d69f7efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:11.407659 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.407642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d890a150-fb5d-466d-ba71-9c08d69f7efe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-282a6587-vlltv\" (UID: \"d890a150-fb5d-466d-ba71-9c08d69f7efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:11.500083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.500001    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:11.772926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.772896    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"]
Sep 15 19:10:11.773816 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:10:11.773788    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd890a150_fb5d_466d_ba71_9c08d69f7efe.slice/crio-18dc40e7ade62136c92cd059924642949c28c0e1adde4f239966fa54616e0cb3 WatchSource:0}: Error finding container 18dc40e7ade62136c92cd059924642949c28c0e1adde4f239966fa54616e0cb3: Status 404 returned error can't find the container with id 18dc40e7ade62136c92cd059924642949c28c0e1adde4f239966fa54616e0cb3
Sep 15 19:10:11.871421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:11.871387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:12.170038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:12.170006    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv" event={"ID":"d890a150-fb5d-466d-ba71-9c08d69f7efe","Type":"ContainerStarted","Data":"e372cf95c46f56cdf88c1886cd269b10acc7b734eb1587b66759aa850e233c04"}
Sep 15 19:10:12.170487 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:12.170068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv" event={"ID":"d890a150-fb5d-466d-ba71-9c08d69f7efe","Type":"ContainerStarted","Data":"18dc40e7ade62136c92cd059924642949c28c0e1adde4f239966fa54616e0cb3"}
Sep 15 19:10:12.184785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:12.184741    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv" podStartSLOduration=1.184725078 podStartE2EDuration="1.184725078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:10:12.183443394 +0000 UTC m=+1164.337811863" watchObservedRunningTime="2026-09-15 19:10:12.184725078 +0000 UTC m=+1164.339093564"
Sep 15 19:10:12.606548 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:12.606509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:13.177739 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:13.177701    2584 generic.go:356] "Generic (PLEG): container finished" podID="d890a150-fb5d-466d-ba71-9c08d69f7efe" containerID="e372cf95c46f56cdf88c1886cd269b10acc7b734eb1587b66759aa850e233c04" exitCode=0
Sep 15 19:10:13.178123 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:13.177795    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv" event={"ID":"d890a150-fb5d-466d-ba71-9c08d69f7efe","Type":"ContainerDied","Data":"e372cf95c46f56cdf88c1886cd269b10acc7b734eb1587b66759aa850e233c04"}
Sep 15 19:10:13.301195 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:13.301157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:13.854349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:13.854313    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:14.320084 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:14.320058    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:14.439935 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:14.439846    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d890a150-fb5d-466d-ba71-9c08d69f7efe-maas-api-service-ca\") pod \"d890a150-fb5d-466d-ba71-9c08d69f7efe\" (UID: \"d890a150-fb5d-466d-ba71-9c08d69f7efe\") "
Sep 15 19:10:14.440196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:14.440172    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d890a150-fb5d-466d-ba71-9c08d69f7efe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d890a150-fb5d-466d-ba71-9c08d69f7efe" (UID: "d890a150-fb5d-466d-ba71-9c08d69f7efe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:10:14.541285 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:14.541252    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d890a150-fb5d-466d-ba71-9c08d69f7efe-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:14.636702 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:14.636656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:15.192459 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:15.192421    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"
Sep 15 19:10:15.192691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:15.192426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv" event={"ID":"d890a150-fb5d-466d-ba71-9c08d69f7efe","Type":"ContainerDied","Data":"18dc40e7ade62136c92cd059924642949c28c0e1adde4f239966fa54616e0cb3"}
Sep 15 19:10:15.192691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:15.192595    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18dc40e7ade62136c92cd059924642949c28c0e1adde4f239966fa54616e0cb3"
Sep 15 19:10:15.353780 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:15.353747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:23.239580 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.239546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:23.338875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.338845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:23.439861 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.439827    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:23.458915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.458879    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"]
Sep 15 19:10:23.462762 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.462738    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-282a6587-vlltv"]
Sep 15 19:10:23.506944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.506912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:23.616246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.616209    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"]
Sep 15 19:10:23.616533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.616506    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" containerID="cri-o://121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55" gracePeriod=30
Sep 15 19:10:23.643874 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.643841    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:23.908676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:23.908592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:24.234877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:24.234794    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d890a150-fb5d-466d-ba71-9c08d69f7efe" path="/var/lib/kubelet/pods/d890a150-fb5d-466d-ba71-9c08d69f7efe/volumes"
Sep 15 19:10:24.273791 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:24.273754    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:24.369692 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:24.369646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:24.473111 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:24.473083    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:24.754134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:24.754102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:25.513363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:25.513331    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:25.777905 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:25.777819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:26.287135 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:26.287042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:26.401710 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:26.401676    2584 patch_prober.go:28] interesting pod/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:10:26.401880 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:26.401745    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:10:26.916851 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:26.916815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:28.876038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.876014    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:10:28.995137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995054    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-data\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995091    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-envoy\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995157    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8wkb\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-kube-api-access-r8wkb\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995181    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-socket\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995216    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-token\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995347    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37e673-7f67-4d1e-af71-d1eb8abac355-istiod-ca-cert\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995389    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:10:28.995560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995442    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-podinfo\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995472    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-certs\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995474    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:10:28.995560 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995521    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-credential-socket\") pod \"dd37e673-7f67-4d1e-af71-d1eb8abac355\" (UID: \"dd37e673-7f67-4d1e-af71-d1eb8abac355\") "
Sep 15 19:10:28.995867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:10:28.995867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995792    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd37e673-7f67-4d1e-af71-d1eb8abac355-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:10:28.996016 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.995999    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:10:28.996187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.996173    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:28.996245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.996194    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37e673-7f67-4d1e-af71-d1eb8abac355-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:28.996245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.996208    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:28.996245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.996221    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:28.996245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.996235    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:28.997584 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.997564    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:10:28.997670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.997638    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:10:28.997913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.997890    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-kube-api-access-r8wkb" (OuterVolumeSpecName: "kube-api-access-r8wkb") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "kube-api-access-r8wkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:10:28.998011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:28.997925    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd37e673-7f67-4d1e-af71-d1eb8abac355" (UID: "dd37e673-7f67-4d1e-af71-d1eb8abac355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:10:29.097802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.097763    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8wkb\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-kube-api-access-r8wkb\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:29.097802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.097796    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:29.097802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.097805    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:29.098042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.097815    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37e673-7f67-4d1e-af71-d1eb8abac355-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:10:29.291993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.291956    2584 generic.go:356] "Generic (PLEG): container finished" podID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerID="121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55" exitCode=0
Sep 15 19:10:29.292175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.292012    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"
Sep 15 19:10:29.292175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.292041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" event={"ID":"dd37e673-7f67-4d1e-af71-d1eb8abac355","Type":"ContainerDied","Data":"121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55"}
Sep 15 19:10:29.292175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.292100    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9" event={"ID":"dd37e673-7f67-4d1e-af71-d1eb8abac355","Type":"ContainerDied","Data":"56ca71c4989802a81c1fd382214bdf56f53eaa8316332e4f69d4c4be9f25c8d1"}
Sep 15 19:10:29.292175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.292122    2584 scope.go:117] "RemoveContainer" containerID="121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55"
Sep 15 19:10:29.302661 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.302646    2584 scope.go:117] "RemoveContainer" containerID="ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"
Sep 15 19:10:29.311225 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.311200    2584 scope.go:117] "RemoveContainer" containerID="121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55"
Sep 15 19:10:29.311641 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:10:29.311522    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55\": container with ID starting with 121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55 not found: ID does not exist" containerID="121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55"
Sep 15 19:10:29.311641 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.311557    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55"} err="failed to get container status \"121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55\": rpc error: code = NotFound desc = could not find container \"121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55\": container with ID starting with 121a2f37741aad90b157b5ca5359f89a980d2e42e96e312b92380fc4188e6c55 not found: ID does not exist"
Sep 15 19:10:29.311641 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.311578    2584 scope.go:117] "RemoveContainer" containerID="ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"
Sep 15 19:10:29.312473 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:10:29.312450    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181\": container with ID starting with ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181 not found: ID does not exist" containerID="ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"
Sep 15 19:10:29.312551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.312478    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181"} err="failed to get container status \"ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181\": rpc error: code = NotFound desc = could not find container \"ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181\": container with ID starting with ec864f0664a030c5d8116ef7a8a918c638807984a02aa350132cbd0ee5692181 not found: ID does not exist"
Sep 15 19:10:29.315009 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.314982    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"]
Sep 15 19:10:29.317760 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:29.317741    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-282a6587-openshift-default-6cfcc7fbf6-9p5g9"]
Sep 15 19:10:30.239503 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:30.239458    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" path="/var/lib/kubelet/pods/dd37e673-7f67-4d1e-af71-d1eb8abac355/volumes"
Sep 15 19:10:31.010739 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:31.010700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:33.435012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:33.434978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:41.503877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:41.503838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:58.831076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:58.831041    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:10:59.389798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:10:59.389765    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:01.166770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.166732    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"]
Sep 15 19:11:01.167408 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167392    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d890a150-fb5d-466d-ba71-9c08d69f7efe" containerName="revoke-keys"
Sep 15 19:11:01.167454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167413    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d890a150-fb5d-466d-ba71-9c08d69f7efe" containerName="revoke-keys"
Sep 15 19:11:01.167454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167434    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy"
Sep 15 19:11:01.167454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167443    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy"
Sep 15 19:11:01.167545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167453    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy"
Sep 15 19:11:01.167545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167462    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy"
Sep 15 19:11:01.167636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167564    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy"
Sep 15 19:11:01.167636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.167578    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d890a150-fb5d-466d-ba71-9c08d69f7efe" containerName="revoke-keys"
Sep 15 19:11:01.172179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.172158    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.174517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.174494    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-900c88\"/\"premium-w0-900c88-kserve-self-signed-certs\""
Sep 15 19:11:01.184521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.184495    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"]
Sep 15 19:11:01.200796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.200763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:01.330047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.330010    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-model-cache\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.330047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.330049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tmp-dir\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.330265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.330161    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-dshm\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.330265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.330195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tls-certs\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.330265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.330252    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf62\" (UniqueName: \"kubernetes.io/projected/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-kube-api-access-6zf62\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.330395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.330277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-home\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.431486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431383    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-model-cache\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.431486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tmp-dir\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.431785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-dshm\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.431785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tls-certs\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.431785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431582    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf62\" (UniqueName: \"kubernetes.io/projected/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-kube-api-access-6zf62\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.431785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431642    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-home\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.432011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-model-cache\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.432011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.431949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tmp-dir\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.432121 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.432027    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-home\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.434293 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.434264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tls-certs\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.434480 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.434462    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-dshm\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.439799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.439773    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf62\" (UniqueName: \"kubernetes.io/projected/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-kube-api-access-6zf62\") pod \"premium-w0-900c88-kserve-78bdb5c5b8-bxw6s\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") " pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.482229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.482201    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:01.758967 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:01.758933    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"]
Sep 15 19:11:01.760088 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:11:01.760061    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefa122f1_f8ef_43db_8a25_b0ee0e8da2c6.slice/crio-056b0a4b65fafb460b32d87080b4770c25127014662cbb94703c37ec826b6728 WatchSource:0}: Error finding container 056b0a4b65fafb460b32d87080b4770c25127014662cbb94703c37ec826b6728: Status 404 returned error can't find the container with id 056b0a4b65fafb460b32d87080b4770c25127014662cbb94703c37ec826b6728
Sep 15 19:11:02.509956 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:02.509919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s" event={"ID":"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6","Type":"ContainerStarted","Data":"e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f"}
Sep 15 19:11:02.510346 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:02.509971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s" event={"ID":"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6","Type":"ContainerStarted","Data":"056b0a4b65fafb460b32d87080b4770c25127014662cbb94703c37ec826b6728"}
Sep 15 19:11:02.526378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:02.526321    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s" podStartSLOduration=1.526304561 podStartE2EDuration="1.526304561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:02.524170917 +0000 UTC m=+1214.678539388" watchObservedRunningTime="2026-09-15 19:11:02.526304561 +0000 UTC m=+1214.680673029"
Sep 15 19:11:04.006198 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:04.006160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:04.123080 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:04.123042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:04.249379 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:04.249348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:11.483058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:11.483017    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:11.483723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:11.483073    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:11.495722 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:11.495684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:11.582357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:11.582328    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:11:20.087600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:20.087565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:23.284410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:23.284377    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:25.712530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:25.712492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:25.785867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:25.785832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:25.891407 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:25.891373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:36.371369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:36.371337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:37.122073 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:37.122035    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:39.507054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:39.507023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:42.932850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:42.932773    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:43.085208 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:43.085175    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:43.185076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:43.184993    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:43.285492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:43.285441    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:43.832605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:43.832569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:43.932935 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:43.932903    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:44.033930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:44.033901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:45.460713 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:45.460681    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:51.323773 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:51.323737    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:51.423258 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:51.423222    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:52.166684 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:52.166605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:57.934487 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:57.934453    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:58.038342 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.038305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:58.135731 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.135702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:58.342329 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.342292    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"]
Sep 15 19:11:58.343103 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.343083    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd37e673-7f67-4d1e-af71-d1eb8abac355" containerName="istio-proxy"
Sep 15 19:11:58.346639 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.346605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.350017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.349994    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a0497c3b-openshift-default-dockercfg-vkmf4\""
Sep 15 19:11:58.357506 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.357484    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"]
Sep 15 19:11:58.369105 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.369079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:58.438971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.438928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-socket\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-token\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439047    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-envoy\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439086    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-credential-socket\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmq4z\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-kube-api-access-kmq4z\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-certs\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439166    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-podinfo\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-data\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.439483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.439292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istiod-ca-cert\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.468569 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.468543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:11:58.540697 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-socket\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.540921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-token\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.540921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-envoy\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.540921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540821    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-credential-socket\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.540921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmq4z\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-kube-api-access-kmq4z\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.540921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-certs\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.540921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540903    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-podinfo\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-data\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.540969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istiod-ca-cert\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.541113    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-socket\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.541237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-credential-socket\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541584 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.541558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-data\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541801 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.541572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-certs\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.541801 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.541697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istiod-ca-cert\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.543147 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.543127    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-envoy\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.543389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.543370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-podinfo\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.548720 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.548697    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-token\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.549804 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.549779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmq4z\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-kube-api-access-kmq4z\") pod \"e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") " pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.659378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.659287    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:11:58.942194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.942164    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"]
Sep 15 19:11:58.942590 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:11:58.942569    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4742bc_ddea_4d7d_9cd6_f54a1cf5d858.slice/crio-8288a0c69de20590c420a175cef1115f2b6ab4f66b5e0b9423cc9a019ebeef4c WatchSource:0}: Error finding container 8288a0c69de20590c420a175cef1115f2b6ab4f66b5e0b9423cc9a019ebeef4c: Status 404 returned error can't find the container with id 8288a0c69de20590c420a175cef1115f2b6ab4f66b5e0b9423cc9a019ebeef4c
Sep 15 19:11:58.944918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.944883    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:11:58.945003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.944951    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:11:58.945003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:58.944987    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:11:59.896983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:59.896944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" event={"ID":"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858","Type":"ContainerStarted","Data":"19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"}
Sep 15 19:11:59.897160 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:59.897038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" event={"ID":"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858","Type":"ContainerStarted","Data":"8288a0c69de20590c420a175cef1115f2b6ab4f66b5e0b9423cc9a019ebeef4c"}
Sep 15 19:11:59.917245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:11:59.917196    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podStartSLOduration=1.9171823639999999 podStartE2EDuration="1.917182364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:11:59.916766555 +0000 UTC m=+1272.071135023" watchObservedRunningTime="2026-09-15 19:11:59.917182364 +0000 UTC m=+1272.071550834"
Sep 15 19:12:00.659729 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:00.659684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:00.664756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:00.664713    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:00.904175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:00.904148    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:00.905135 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:00.905115    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:03.510559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:03.510511    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:04.005809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:04.005774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:04.105581 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:04.105551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:04.454011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:04.453930    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:05.142863 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:05.142831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:05.987770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:05.987734    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:07.153589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:07.153556    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:07.223518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:07.223483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:07.643761 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:07.643729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:08.322997 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:08.322961    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:26.107206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:26.107167    2584 generic.go:356] "Generic (PLEG): container finished" podID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerID="19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4" exitCode=0
Sep 15 19:12:26.107687 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:26.107240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" event={"ID":"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858","Type":"ContainerDied","Data":"19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"}
Sep 15 19:12:26.107755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:26.107712    2584 scope.go:117] "RemoveContainer" containerID="19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"
Sep 15 19:12:26.108956 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:26.108926    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:12:26.109021 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:26.109000    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:12:26.109056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:26.109032    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:12:27.116305 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:27.116271    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" event={"ID":"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858","Type":"ContainerStarted","Data":"4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39"}
Sep 15 19:12:27.660277 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:27.660233    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:27.661334 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:27.661314    2584 patch_prober.go:28] interesting pod/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 19:12:27.661393 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:27.661375    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 19:12:28.659725 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:28.659689    2584 patch_prober.go:28] interesting pod/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 19:12:28.660325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:28.659755    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 19:12:28.660325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:28.660155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:28.818826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:28.818795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:29.443736 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:29.443698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:29.537007 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:29.536974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:29.660742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:29.660704    2584 patch_prober.go:28] interesting pod/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 19:12:29.661107 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:29.660778    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 19:12:29.850797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:29.850745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:30.180793 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:30.180682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:30.660480 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:30.660446    2584 patch_prober.go:28] interesting pod/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 19:12:30.660692 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:30.660513    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 19:12:30.848709 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:30.848666    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:30.905590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:30.904703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:31.625127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:31.625090    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:31.660487 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:31.660449    2584 patch_prober.go:28] interesting pod/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 15 19:12:31.660712 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:31.660513    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 15 19:12:32.247426 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:32.247384    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:32.664135 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:32.664103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:32.665195 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:32.665171    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:35.459093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.459054    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:35.520736 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.520702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:35.777505 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.777470    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"]
Sep 15 19:12:35.781163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.781144    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:35.783794 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.783757    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:12:35.783932 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.783756    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:12:35.787311 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.787290    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"]
Sep 15 19:12:35.852982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.852945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2\" (UID: \"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:35.954456 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.954414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2\" (UID: \"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:35.955095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:35.955075    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2\" (UID: \"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:36.095225 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:36.095145    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:36.371446 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:36.371410    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"]
Sep 15 19:12:36.372262 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:12:36.372230    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c9ad3b_7b0a_43da_bdc5_5a03d9f36ad8.slice/crio-8b7641e69392da0d1fc64f3a5d25ceeee120b60cbfdaa92407b7dcf71b5f03c2 WatchSource:0}: Error finding container 8b7641e69392da0d1fc64f3a5d25ceeee120b60cbfdaa92407b7dcf71b5f03c2: Status 404 returned error can't find the container with id 8b7641e69392da0d1fc64f3a5d25ceeee120b60cbfdaa92407b7dcf71b5f03c2
Sep 15 19:12:36.686119 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:36.686085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:37.202767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:37.202732    2584 generic.go:356] "Generic (PLEG): container finished" podID="80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8" containerID="c42e24fea382b2353154e06c9ae3025f61a831811305e193653805cd7af30675" exitCode=0
Sep 15 19:12:37.202967 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:37.202822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2" event={"ID":"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8","Type":"ContainerDied","Data":"c42e24fea382b2353154e06c9ae3025f61a831811305e193653805cd7af30675"}
Sep 15 19:12:37.202967 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:37.202922    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2" event={"ID":"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8","Type":"ContainerStarted","Data":"8b7641e69392da0d1fc64f3a5d25ceeee120b60cbfdaa92407b7dcf71b5f03c2"}
Sep 15 19:12:37.215181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:37.215153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:38.023978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:38.023940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:38.360703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:38.360675    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:38.481312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:38.481271    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8-maas-api-service-ca\") pod \"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8\" (UID: \"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8\") "
Sep 15 19:12:38.481730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:38.481707    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8" (UID: "80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:12:38.582952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:38.582921    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:38.684105 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:38.684020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:39.217770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:39.217739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"
Sep 15 19:12:39.217770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:39.217746    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2" event={"ID":"80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8","Type":"ContainerDied","Data":"8b7641e69392da0d1fc64f3a5d25ceeee120b60cbfdaa92407b7dcf71b5f03c2"}
Sep 15 19:12:39.218209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:39.217794    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7641e69392da0d1fc64f3a5d25ceeee120b60cbfdaa92407b7dcf71b5f03c2"
Sep 15 19:12:39.402415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:39.402380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:40.026499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:40.026464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:43.959133 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:43.959099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:46.382601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.382568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:46.447046 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.447010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:46.522873 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.522840    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:46.565407 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.565367    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"]
Sep 15 19:12:46.568161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.568134    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a0497c3b-r7lj2"]
Sep 15 19:12:46.621442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.621401    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:46.947164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:46.947129    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:47.048691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:47.048639    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:47.769014 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:47.768977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:48.234829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:48.234745    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8" path="/var/lib/kubelet/pods/80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8/volumes"
Sep 15 19:12:48.297806 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:48.297776    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:49.027102 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:49.027051    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:49.647087 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:49.647056    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:52.674858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:52.674816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:52.738290 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:52.738248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:52.916380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:52.916351    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"]
Sep 15 19:12:52.916633 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:52.916592    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy" containerID="cri-o://4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39" gracePeriod=30
Sep 15 19:12:52.945913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:52.945793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:53.236936 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:53.236848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:53.337120 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:53.337085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:53.438621 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:53.438571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:58.175514 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.175482    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:58.303629 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303583    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmq4z\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-kube-api-access-kmq4z\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.303804 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303683    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-credential-socket\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.303804 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303721    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-certs\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.303804 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303752    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-socket\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.303804 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303775    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istiod-ca-cert\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.304033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303815    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-envoy\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.304033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303860    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-data\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.304033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.303937    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-podinfo\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.304033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304003    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:12:58.304033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304015    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:12:58.304304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304010    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-token\") pod \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\" (UID: \"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858\") "
Sep 15 19:12:58.304304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304059    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:12:58.304304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304182    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:12:58.304518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304489    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:12:58.304593 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304572    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.304678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304592    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.304678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304605    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.304678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.304647    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.306143 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.306110    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:12:58.306264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.306246    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:12:58.306310 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.306271    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-kube-api-access-kmq4z" (OuterVolumeSpecName: "kube-api-access-kmq4z") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "kube-api-access-kmq4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:12:58.306478 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.306455    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" (UID: "8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:12:58.366977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.366939    2584 generic.go:356] "Generic (PLEG): container finished" podID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerID="4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39" exitCode=0
Sep 15 19:12:58.367134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.366982    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"
Sep 15 19:12:58.367134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.367019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" event={"ID":"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858","Type":"ContainerDied","Data":"4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39"}
Sep 15 19:12:58.367134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.367069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s" event={"ID":"8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858","Type":"ContainerDied","Data":"8288a0c69de20590c420a175cef1115f2b6ab4f66b5e0b9423cc9a019ebeef4c"}
Sep 15 19:12:58.367134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.367090    2584 scope.go:117] "RemoveContainer" containerID="4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39"
Sep 15 19:12:58.384531 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.384514    2584 scope.go:117] "RemoveContainer" containerID="19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"
Sep 15 19:12:58.395072 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.395046    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"]
Sep 15 19:12:58.396321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.396297    2584 scope.go:117] "RemoveContainer" containerID="4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39"
Sep 15 19:12:58.396658 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:12:58.396632    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39\": container with ID starting with 4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39 not found: ID does not exist" containerID="4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39"
Sep 15 19:12:58.396710 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.396668    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39"} err="failed to get container status \"4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39\": rpc error: code = NotFound desc = could not find container \"4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39\": container with ID starting with 4892a86018cdd6db7fe5bc56a5fa3481d66d01304ad936a0d266d91f3deeaf39 not found: ID does not exist"
Sep 15 19:12:58.396710 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.396686    2584 scope.go:117] "RemoveContainer" containerID="19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"
Sep 15 19:12:58.396969 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:12:58.396953    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4\": container with ID starting with 19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4 not found: ID does not exist" containerID="19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"
Sep 15 19:12:58.397007 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.396975    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4"} err="failed to get container status \"19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4\": rpc error: code = NotFound desc = could not find container \"19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4\": container with ID starting with 19cebf5e9bc6bfddb2481794adcd918f42c2c9351ebece01d7dc0df3bfdb37e4 not found: ID does not exist"
Sep 15 19:12:58.398714 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.398694    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a0497c3b-openshift-default-5c96f48cc6-2d48s"]
Sep 15 19:12:58.405597 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.405580    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmq4z\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-kube-api-access-kmq4z\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.405671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.405600    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.405671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.405638    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.405671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.405649    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:58.405671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:58.405656    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:12:59.449783 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:59.449754    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:59.551215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:59.551179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:59.653344 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:59.653315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:12:59.753094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:12:59.753016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:00.235330 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:00.235297    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" path="/var/lib/kubelet/pods/8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858/volumes"
Sep 15 19:13:05.403268 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:05.403234    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:05.503792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:05.503758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:06.281234 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.281199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:06.511147 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.511119    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:06.580474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.580394    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"]
Sep 15 19:13:06.580937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.580915    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy"
Sep 15 19:13:06.580937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.580938    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy"
Sep 15 19:13:06.581094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.580960    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy"
Sep 15 19:13:06.581094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.580968    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy"
Sep 15 19:13:06.581094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.580995    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8" containerName="revoke-keys"
Sep 15 19:13:06.581094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.581004    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8" containerName="revoke-keys"
Sep 15 19:13:06.581094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.581088    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy"
Sep 15 19:13:06.581287 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.581102    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c9ad3b-7b0a-43da-bdc5-5a03d9f36ad8" containerName="revoke-keys"
Sep 15 19:13:06.581442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.581422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:06.585719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.585703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.588048 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.588029    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-362ea4-openshift-default-dockercfg-h67hr\""
Sep 15 19:13:06.598825 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.598801    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"]
Sep 15 19:13:06.689813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.689784    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zgf\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-kube-api-access-58zgf\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.689958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.689835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-credential-socket\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.689958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.689918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-socket\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.689958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.689950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istiod-ca-cert\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.690069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.689987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-certs\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.690069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.690016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-podinfo\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.690069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.690040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-token\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.690190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.690069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-envoy\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.690190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.690107    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-data\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-socket\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istiod-ca-cert\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791745 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-certs\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791745 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-podinfo\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791745 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-token\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791907 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-envoy\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791907 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-data\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.791907 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zgf\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-kube-api-access-58zgf\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.792023 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-certs\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.792023 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.791949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-credential-socket\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.792160 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.792136    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-socket\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.792288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.792271    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istiod-ca-cert\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.792421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.792404    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-credential-socket\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.792536 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.792514    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-data\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.794142 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.794119    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-envoy\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.794355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.794335    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-podinfo\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.799412 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.799387    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-token\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.799796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.799774    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zgf\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-kube-api-access-58zgf\") pod \"e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") " pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:06.855622 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.855549    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:06.897019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:06.896993    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:07.193670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.193641    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"]
Sep 15 19:13:07.194345 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:13:07.194308    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb4cb51_3ed1_43bb_b281_c35dfb310492.slice/crio-acb90759c7e816fb10ae65681e3b172b392f5c00618296063b8bc70be506321c WatchSource:0}: Error finding container acb90759c7e816fb10ae65681e3b172b392f5c00618296063b8bc70be506321c: Status 404 returned error can't find the container with id acb90759c7e816fb10ae65681e3b172b392f5c00618296063b8bc70be506321c
Sep 15 19:13:07.197243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.197200    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:07.197376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.197287    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:07.197376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.197327    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:07.429160 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.429115    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" event={"ID":"fdb4cb51-3ed1-43bb-b281-c35dfb310492","Type":"ContainerStarted","Data":"39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"}
Sep 15 19:13:07.429330 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.429179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" event={"ID":"fdb4cb51-3ed1-43bb-b281-c35dfb310492","Type":"ContainerStarted","Data":"acb90759c7e816fb10ae65681e3b172b392f5c00618296063b8bc70be506321c"}
Sep 15 19:13:07.448403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.448317    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podStartSLOduration=1.4483022349999999 podStartE2EDuration="1.448302235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:07.445671925 +0000 UTC m=+1339.600040394" watchObservedRunningTime="2026-09-15 19:13:07.448302235 +0000 UTC m=+1339.602670703"
Sep 15 19:13:07.897562 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:07.897529    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:08.622653 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:08.622592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:08.902276 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:08.902188    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:09.426930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:09.426894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:09.444649 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:09.444604    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:09.445449 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:09.445428    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:09.523595 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:09.523563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:10.232332 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:10.232299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:11.125740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:11.125656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:11.856716 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:11.856681    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:12.429983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:12.429940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:12.530834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:12.530802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:12.866446 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:12.866409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:12.971808 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:12.971773    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:13.072487 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:13.072455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:13.151051 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:13.150977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:13.532813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:13.532774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:14.243209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:14.243177    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:14.930954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:14.930918    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:34.282303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.282269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:34.559974 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.559888    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:34.660998 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.660963    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:34.858760 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.858682    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"]
Sep 15 19:13:34.859588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.859520    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4742bc-ddea-4d7d-9cd6-f54a1cf5d858" containerName="istio-proxy"
Sep 15 19:13:34.863070 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.863048    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.865802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.865781    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-1b0e3f-openshift-default-dockercfg-dfn9w\""
Sep 15 19:13:34.875019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.874993    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"]
Sep 15 19:13:34.882416 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.882393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:34.980633 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.980548    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-certs\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.980838 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.980672    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-token\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.980838 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.980787    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-credential-socket\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.980958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.980873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-data\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.980958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.980935    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwkk\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-kube-api-access-snwkk\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.981069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.981013    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-socket\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.981069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.981037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-envoy\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.981178 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.981131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istiod-ca-cert\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:34.981233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:34.981191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-podinfo\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.082949 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.082912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-credential-socket\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.082981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-data\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083023    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwkk\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-kube-api-access-snwkk\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083068    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-socket\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-envoy\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istiod-ca-cert\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-podinfo\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-certs\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-token\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-credential-socket\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-data\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083540    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-certs\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.083922 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.083900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-socket\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.084205 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.084184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istiod-ca-cert\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.085444 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.085423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-envoy\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.085764 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.085745    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-podinfo\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.092667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.092647    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-token\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.092991 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.092971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwkk\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-kube-api-access-snwkk\") pod \"e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") " pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.125640 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.125505    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:35.179872 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.179846    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:35.348937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.348887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:35.486239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.486211    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"]
Sep 15 19:13:35.488212 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:13:35.488184    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4f654e_81b8_4480_a34c_cdb51f2831d3.slice/crio-26fccc71c5f12af82120aab4fb7458d4064e34b775e8b4e51524c13a5f3ef8e7 WatchSource:0}: Error finding container 26fccc71c5f12af82120aab4fb7458d4064e34b775e8b4e51524c13a5f3ef8e7: Status 404 returned error can't find the container with id 26fccc71c5f12af82120aab4fb7458d4064e34b775e8b4e51524c13a5f3ef8e7
Sep 15 19:13:35.490548 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.490515    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:35.490655 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.490582    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:35.490655 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.490641    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:35.632488 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.632443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" event={"ID":"fe4f654e-81b8-4480-a34c-cdb51f2831d3","Type":"ContainerStarted","Data":"1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"}
Sep 15 19:13:35.632685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.632517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" event={"ID":"fe4f654e-81b8-4480-a34c-cdb51f2831d3","Type":"ContainerStarted","Data":"26fccc71c5f12af82120aab4fb7458d4064e34b775e8b4e51524c13a5f3ef8e7"}
Sep 15 19:13:35.655756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:35.655697    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podStartSLOduration=1.655676187 podStartE2EDuration="1.655676187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:35.649872071 +0000 UTC m=+1367.804240540" watchObservedRunningTime="2026-09-15 19:13:35.655676187 +0000 UTC m=+1367.810044656"
Sep 15 19:13:36.180577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:36.180545    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:36.182219 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:36.182186    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:13:36.182365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:36.182237    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:13:36.189844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:36.189808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:36.989229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:36.989201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:37.185832 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:37.185803    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:37.527237 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:37.527202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:37.626465 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:37.626404    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:37.655648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:37.655593    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:37.658190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:37.657972    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:13:38.420518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:38.420472    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:39.160385 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:39.160355    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:39.262243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:39.262214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:39.461269 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:39.461183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:39.698256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:39.698228    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:39.780521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:39.780495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:40.140257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.140178    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:40.269894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.269867    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": read tcp 10.133.0.2:58780->10.133.0.75:15021: read: connection reset by peer" start-of-body=
Sep 15 19:13:40.270078 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.269930    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": read tcp 10.133.0.2:58780->10.133.0.75:15021: read: connection reset by peer"
Sep 15 19:13:40.676455 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.676425    2584 generic.go:356] "Generic (PLEG): container finished" podID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerID="39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60" exitCode=0
Sep 15 19:13:40.676948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.676465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" event={"ID":"fdb4cb51-3ed1-43bb-b281-c35dfb310492","Type":"ContainerDied","Data":"39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"}
Sep 15 19:13:40.677056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.677036    2584 scope.go:117] "RemoveContainer" containerID="39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"
Sep 15 19:13:40.678303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.678265    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:40.678394 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.678343    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:40.678394 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:40.678382    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:13:41.029607 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:41.029565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:41.685197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:41.685163    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" event={"ID":"fdb4cb51-3ed1-43bb-b281-c35dfb310492","Type":"ContainerStarted","Data":"77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e"}
Sep 15 19:13:41.737637 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:41.737577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:41.897775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:41.897741    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:41.897946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:41.897883    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 19:13:41.897946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:41.897935    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 19:13:42.488663 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:42.488603    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:42.897942 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:42.897907    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 19:13:42.898357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:42.897961    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 19:13:43.897402 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:43.897364    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 19:13:43.897596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:43.897432    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 19:13:44.897969 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:44.897930    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 19:13:44.898375 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:44.897994    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 19:13:45.897370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:45.897330    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 19:13:45.897552 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:45.897393    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 19:13:46.897914 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:46.897881    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 15 19:13:46.898280 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:46.897965    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 15 19:13:47.361763 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.361730    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"]
Sep 15 19:13:47.365162 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.365136    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.368458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.368433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"default-dockercfg-cv7tx\""
Sep 15 19:13:47.368578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.368467    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"kube-root-ca.crt\""
Sep 15 19:13:47.368578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.368433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"distinct-2-w3-f0ad35-kserve-self-signed-certs\""
Sep 15 19:13:47.368578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.368527    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f0ad35\"/\"openshift-service-ca.crt\""
Sep 15 19:13:47.374376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.373973    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"]
Sep 15 19:13:47.519004 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.518967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2kq\" (UniqueName: \"kubernetes.io/projected/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-kube-api-access-pb2kq\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.519182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.519014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-home\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.519182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.519049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-model-cache\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.519264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.519173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tmp-dir\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.519264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.519233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-dshm\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.519264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.519251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tls-certs\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.620901 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.620793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2kq\" (UniqueName: \"kubernetes.io/projected/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-kube-api-access-pb2kq\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.620901 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.620845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-home\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.620901 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.620876    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-model-cache\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.621202 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.620957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tmp-dir\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.621202 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.621019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-dshm\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.621202 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.621045    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tls-certs\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.621372 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.621294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-home\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.621725 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.621699    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tmp-dir\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.621858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.621744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-model-cache\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.623803 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.623783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-dshm\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.624103 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.624080    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tls-certs\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.628193 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.628163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2kq\" (UniqueName: \"kubernetes.io/projected/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-kube-api-access-pb2kq\") pod \"distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") " pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.666976 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.666941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:47.676915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.676892    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:47.901567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.901483    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:47.901921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.901890    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:47.902501 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.902481    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:13:47.952184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:47.952081    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"]
Sep 15 19:13:47.953407 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:13:47.953382    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d83151e_d37c_42e9_b9f4_17411dbe0ca3.slice/crio-7eabb0dbb8cdfa640dd45012555c7ba8050de58b45b89069dd33e37aec3f82d2 WatchSource:0}: Error finding container 7eabb0dbb8cdfa640dd45012555c7ba8050de58b45b89069dd33e37aec3f82d2: Status 404 returned error can't find the container with id 7eabb0dbb8cdfa640dd45012555c7ba8050de58b45b89069dd33e37aec3f82d2
Sep 15 19:13:48.733867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:48.733778    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z" event={"ID":"3d83151e-d37c-42e9-b9f4-17411dbe0ca3","Type":"ContainerStarted","Data":"74c738aec27eb0fed55d6ff308f8adb9edf06e2d72816a64c39518775eb75177"}
Sep 15 19:13:48.733867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:48.733825    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z" event={"ID":"3d83151e-d37c-42e9-b9f4-17411dbe0ca3","Type":"ContainerStarted","Data":"7eabb0dbb8cdfa640dd45012555c7ba8050de58b45b89069dd33e37aec3f82d2"}
Sep 15 19:13:48.752234 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:48.752182    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z" podStartSLOduration=1.752166054 podStartE2EDuration="1.752166054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:13:48.7495543 +0000 UTC m=+1380.903922769" watchObservedRunningTime="2026-09-15 19:13:48.752166054 +0000 UTC m=+1380.906534522"
Sep 15 19:13:53.943898 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:53.943861    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:54.041134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:54.041092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:13:57.677133 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:57.677091    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:57.677507 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:57.677150    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:57.689938 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:57.689905    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:13:57.803977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:13:57.803941    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:14:01.215303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:01.215270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:01.894805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:01.894770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:02.303010 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:02.302978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:02.438867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:02.438838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:03.254122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:03.254079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:04.024588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.024558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:04.781935 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.781900    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:04.847802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.847707    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerID="1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d" exitCode=0
Sep 15 19:14:04.847802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.847800    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" event={"ID":"fe4f654e-81b8-4480-a34c-cdb51f2831d3","Type":"ContainerDied","Data":"1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"}
Sep 15 19:14:04.848414 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.848354    2584 scope.go:117] "RemoveContainer" containerID="1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"
Sep 15 19:14:04.849666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.849631    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:04.849726 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.849708    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:04.849767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:04.849743    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:05.565641 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:05.565570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:05.857557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:05.857469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" event={"ID":"fe4f654e-81b8-4480-a34c-cdb51f2831d3","Type":"ContainerStarted","Data":"e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40"}
Sep 15 19:14:06.180648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:06.180549    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:14:06.181474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:06.181451    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:14:06.181530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:06.181512    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:14:07.180732 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:07.180695    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:14:07.181112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:07.180761    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:14:07.656183 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:07.656128    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:14:08.180225 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:08.180193    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:14:08.180398 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:08.180246    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:14:09.180374 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:09.180335    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:14:09.180846 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:09.180420    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:14:10.180767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:10.180728    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:14:10.181187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:10.180785    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:14:11.180820 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.180775    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 15 19:14:11.181236 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.180864    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 15 19:14:11.490054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.489982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:11.742777 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.742695    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"]
Sep 15 19:14:11.746115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.746093    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.748385 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.748366    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-17c562de-openshift-default-dockercfg-h4rsj\""
Sep 15 19:14:11.764559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.764526    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"]
Sep 15 19:14:11.773534 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.773508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:11.811585 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-envoy\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.811750 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811688    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istiod-ca-cert\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.811750 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811722    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-credential-socket\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.811875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811763    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-token\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.811875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811788    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-podinfo\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.811875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-data\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.812035 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811923    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-certs\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.812035 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97mbw\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-kube-api-access-97mbw\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.812035 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.811992    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-socket\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912736 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-envoy\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istiod-ca-cert\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-credential-socket\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912835    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-token\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-podinfo\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-data\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.912957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-certs\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913299 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.912990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97mbw\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-kube-api-access-97mbw\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913299 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.913011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-socket\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913299 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.913244    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-credential-socket\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913468 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.913392    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-certs\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913468 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.913453    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-socket\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913582 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.913497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-data\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.913652 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.913585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istiod-ca-cert\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.915221 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.915199    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-envoy\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.915335 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.915291    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-podinfo\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.920502 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.920476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-token\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:11.921058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:11.921036    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97mbw\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-kube-api-access-97mbw\") pod \"e2e-mt-17c562de-openshift-default-88f855448-8xcsb\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") " pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:12.047485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.047452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:12.060547 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.060516    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:12.184136 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.184105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:14:12.185198 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.185176    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:14:12.339805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.339778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"]
Sep 15 19:14:12.340587 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:14:12.340559    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecb75a1_deb7_4b19_aa84_d3e71542c97e.slice/crio-a9742a6d75437df2c66192044dc6f1a5198ce8fd5136e7e4945794a895e3f3f0 WatchSource:0}: Error finding container a9742a6d75437df2c66192044dc6f1a5198ce8fd5136e7e4945794a895e3f3f0: Status 404 returned error can't find the container with id a9742a6d75437df2c66192044dc6f1a5198ce8fd5136e7e4945794a895e3f3f0
Sep 15 19:14:12.343093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.343049    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:12.343192 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.343172    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:12.343235 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.343224    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:12.910239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.910203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" event={"ID":"5ecb75a1-deb7-4b19-aa84-d3e71542c97e","Type":"ContainerStarted","Data":"da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"}
Sep 15 19:14:12.910409 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.910262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" event={"ID":"5ecb75a1-deb7-4b19-aa84-d3e71542c97e","Type":"ContainerStarted","Data":"a9742a6d75437df2c66192044dc6f1a5198ce8fd5136e7e4945794a895e3f3f0"}
Sep 15 19:14:12.932918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:12.932874    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podStartSLOduration=1.932858322 podStartE2EDuration="1.932858322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:14:12.929599311 +0000 UTC m=+1405.083967790" watchObservedRunningTime="2026-09-15 19:14:12.932858322 +0000 UTC m=+1405.087226856"
Sep 15 19:14:13.061316 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:13.061279    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:13.066285 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:13.066259    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:13.916595 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:13.916570    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:13.917559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:13.917542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:14.235387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:14.235304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:14.784929 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:14.784899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:14.881983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:14.881951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:15.958732 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:15.958690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:16.740976 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:16.740916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:17.515992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:17.515960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:18.436437 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:18.436403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:19.152626 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:19.152581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:19.927006 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:19.926974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:20.028301 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:20.028266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:20.360807 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:20.360778    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:20.500746 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:20.500716    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:20.599756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:20.599721    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:24.078430 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.078394    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"]
Sep 15 19:14:24.082227 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.082211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:24.085153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.085128    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:14:24.085788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.085764    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:14:24.089035 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.089012    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"]
Sep 15 19:14:24.256086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.256054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e278d84-55d7-4499-9de1-4eb4c361d678-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7\" (UID: \"0e278d84-55d7-4499-9de1-4eb4c361d678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:24.336141 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.336065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:24.357645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.357588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e278d84-55d7-4499-9de1-4eb4c361d678-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7\" (UID: \"0e278d84-55d7-4499-9de1-4eb4c361d678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:24.358225 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.358204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e278d84-55d7-4499-9de1-4eb4c361d678-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7\" (UID: \"0e278d84-55d7-4499-9de1-4eb4c361d678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:24.395759 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.395737    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:24.679332 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.679305    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"]
Sep 15 19:14:24.680152 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:14:24.680125    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e278d84_55d7_4499_9de1_4eb4c361d678.slice/crio-c750ecc79a4be1c91586e6d9f9884bed7eb5eff87742dd2abb7d90bd4b55b4fb WatchSource:0}: Error finding container c750ecc79a4be1c91586e6d9f9884bed7eb5eff87742dd2abb7d90bd4b55b4fb: Status 404 returned error can't find the container with id c750ecc79a4be1c91586e6d9f9884bed7eb5eff87742dd2abb7d90bd4b55b4fb
Sep 15 19:14:24.996415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:24.996333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7" event={"ID":"0e278d84-55d7-4499-9de1-4eb4c361d678","Type":"ContainerStarted","Data":"c750ecc79a4be1c91586e6d9f9884bed7eb5eff87742dd2abb7d90bd4b55b4fb"}
Sep 15 19:14:25.238130 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:25.238102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:25.879969 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:25.879940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:26.005056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:26.005021    2584 generic.go:356] "Generic (PLEG): container finished" podID="0e278d84-55d7-4499-9de1-4eb4c361d678" containerID="17eab260613e0707385a85f9d5106e143094549b42e2167931049401ff983b38" exitCode=0
Sep 15 19:14:26.005232 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:26.005075    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7" event={"ID":"0e278d84-55d7-4499-9de1-4eb4c361d678","Type":"ContainerDied","Data":"17eab260613e0707385a85f9d5106e143094549b42e2167931049401ff983b38"}
Sep 15 19:14:26.459005 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:26.458919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:27.116755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:27.116726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:27.148509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:27.148484    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:27.291376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:27.291290    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e278d84-55d7-4499-9de1-4eb4c361d678-maas-api-service-ca\") pod \"0e278d84-55d7-4499-9de1-4eb4c361d678\" (UID: \"0e278d84-55d7-4499-9de1-4eb4c361d678\") "
Sep 15 19:14:27.291664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:27.291644    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e278d84-55d7-4499-9de1-4eb4c361d678-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e278d84-55d7-4499-9de1-4eb4c361d678" (UID: "0e278d84-55d7-4499-9de1-4eb4c361d678"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:14:27.392433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:27.392401    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e278d84-55d7-4499-9de1-4eb4c361d678-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:27.850668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:27.850599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:28.019025 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:28.018992    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7" event={"ID":"0e278d84-55d7-4499-9de1-4eb4c361d678","Type":"ContainerDied","Data":"c750ecc79a4be1c91586e6d9f9884bed7eb5eff87742dd2abb7d90bd4b55b4fb"}
Sep 15 19:14:28.019025 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:28.019017    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"
Sep 15 19:14:28.019230 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:28.019039    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c750ecc79a4be1c91586e6d9f9884bed7eb5eff87742dd2abb7d90bd4b55b4fb"
Sep 15 19:14:28.649895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:28.649865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:29.313811 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:29.313780    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:29.997392 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:29.997353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:32.877324 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:32.877291    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:34.766482 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:34.766446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:34.866452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:34.866410    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:34.959898 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:34.959868    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"]
Sep 15 19:14:34.965490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:34.965451    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9de44d-jtrm7"]
Sep 15 19:14:36.239595 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:36.239550    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e278d84-55d7-4499-9de1-4eb4c361d678" path="/var/lib/kubelet/pods/0e278d84-55d7-4499-9de1-4eb4c361d678/volumes"
Sep 15 19:14:36.402241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:36.402206    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:36.505721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:36.505680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:36.851441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:36.851345    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:37.505382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:37.505350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:38.149793 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:38.149752    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:40.056911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:40.056882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:40.156847 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:40.156816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:40.287344 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:40.287306    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"]
Sep 15 19:14:40.382576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:40.382491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:40.795083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:40.795045    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:41.499577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:41.499547    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:42.046993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.046951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:42.123839 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.123754    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerID="da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be" exitCode=0
Sep 15 19:14:42.123839 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.123829    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" event={"ID":"5ecb75a1-deb7-4b19-aa84-d3e71542c97e","Type":"ContainerDied","Data":"da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"}
Sep 15 19:14:42.124363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.124345    2584 scope.go:117] "RemoveContainer" containerID="da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"
Sep 15 19:14:42.125668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.125636    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:42.125753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.125704    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:42.125753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.125736    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:14:42.826364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:42.826321    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:43.129202 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:43.129114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:43.153445 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:43.153395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" event={"ID":"5ecb75a1-deb7-4b19-aa84-d3e71542c97e","Type":"ContainerStarted","Data":"f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a"}
Sep 15 19:14:43.917204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:43.917178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:44.060888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:44.060855    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:44.061725 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:44.061703    2584 patch_prober.go:28] interesting pod/e2e-mt-17c562de-openshift-default-88f855448-8xcsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 15 19:14:44.061790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:44.061750    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 15 19:14:45.011171 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:45.011130    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:45.061421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:45.061391    2584 patch_prober.go:28] interesting pod/e2e-mt-17c562de-openshift-default-88f855448-8xcsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 15 19:14:45.061581 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:45.061442    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 15 19:14:45.112528 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:45.112500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:14:46.061312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:46.061275    2584 patch_prober.go:28] interesting pod/e2e-mt-17c562de-openshift-default-88f855448-8xcsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 15 19:14:46.061711 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:46.061343    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 15 19:14:47.061044 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:47.061008    2584 patch_prober.go:28] interesting pod/e2e-mt-17c562de-openshift-default-88f855448-8xcsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 15 19:14:47.061217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:47.061067    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 15 19:14:48.061462 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:48.061405    2584 patch_prober.go:28] interesting pod/e2e-mt-17c562de-openshift-default-88f855448-8xcsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Sep 15 19:14:48.061926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:48.061505    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 15 19:14:48.478586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:48.478489    2584 scope.go:117] "RemoveContainer" containerID="f54342b640c550d9a6b86b05571f5ab0c41a16f42e36230674fd2d046bda7efa"
Sep 15 19:14:49.065460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:49.065431    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:49.066471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:49.066450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:14:55.307148 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.307106    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55" podUID="d5fa3580-f5f8-4ecc-97f7-b491ca162576" containerName="main" containerID="cri-o://aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876" gracePeriod=45
Sep 15 19:14:55.560426 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.560364    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:14:55.705844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.705797    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nsfv\" (UniqueName: \"kubernetes.io/projected/d5fa3580-f5f8-4ecc-97f7-b491ca162576-kube-api-access-7nsfv\") pod \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") "
Sep 15 19:14:55.706039 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.705883    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-model-cache\") pod \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") "
Sep 15 19:14:55.706039 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.705976    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-dshm\") pod \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") "
Sep 15 19:14:55.706039 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706018    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tls-certs\") pod \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") "
Sep 15 19:14:55.706217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706064    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tmp-dir\") pod \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") "
Sep 15 19:14:55.706217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706073    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5fa3580-f5f8-4ecc-97f7-b491ca162576" (UID: "d5fa3580-f5f8-4ecc-97f7-b491ca162576"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:55.706217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706101    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-home\") pod \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\" (UID: \"d5fa3580-f5f8-4ecc-97f7-b491ca162576\") "
Sep 15 19:14:55.706382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706301    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5fa3580-f5f8-4ecc-97f7-b491ca162576" (UID: "d5fa3580-f5f8-4ecc-97f7-b491ca162576"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:55.706429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706377    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-home" (OuterVolumeSpecName: "home") pod "d5fa3580-f5f8-4ecc-97f7-b491ca162576" (UID: "d5fa3580-f5f8-4ecc-97f7-b491ca162576"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:55.706576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706556    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:55.706576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706577    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:55.706772 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.706588    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:55.708095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.708064    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-dshm" (OuterVolumeSpecName: "dshm") pod "d5fa3580-f5f8-4ecc-97f7-b491ca162576" (UID: "d5fa3580-f5f8-4ecc-97f7-b491ca162576"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:14:55.708095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.708073    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fa3580-f5f8-4ecc-97f7-b491ca162576-kube-api-access-7nsfv" (OuterVolumeSpecName: "kube-api-access-7nsfv") pod "d5fa3580-f5f8-4ecc-97f7-b491ca162576" (UID: "d5fa3580-f5f8-4ecc-97f7-b491ca162576"). InnerVolumeSpecName "kube-api-access-7nsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:14:55.708233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.708154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5fa3580-f5f8-4ecc-97f7-b491ca162576" (UID: "d5fa3580-f5f8-4ecc-97f7-b491ca162576"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:14:55.807997 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.807958    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nsfv\" (UniqueName: \"kubernetes.io/projected/d5fa3580-f5f8-4ecc-97f7-b491ca162576-kube-api-access-7nsfv\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:55.807997 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.807994    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5fa3580-f5f8-4ecc-97f7-b491ca162576-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:55.807997 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:55.808004    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5fa3580-f5f8-4ecc-97f7-b491ca162576-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:14:56.245108 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.245066    2584 generic.go:356] "Generic (PLEG): container finished" podID="d5fa3580-f5f8-4ecc-97f7-b491ca162576" containerID="aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876" exitCode=0
Sep 15 19:14:56.245252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.245109    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"
Sep 15 19:14:56.245252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.245124    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55" event={"ID":"d5fa3580-f5f8-4ecc-97f7-b491ca162576","Type":"ContainerDied","Data":"aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876"}
Sep 15 19:14:56.245252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.245162    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55" event={"ID":"d5fa3580-f5f8-4ecc-97f7-b491ca162576","Type":"ContainerDied","Data":"fe4b2d10a23e580de6a8d57de7ba83626a3560fa0b6893ca6e1540a2900f2a88"}
Sep 15 19:14:56.245252 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.245182    2584 scope.go:117] "RemoveContainer" containerID="aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876"
Sep 15 19:14:56.255026 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.255009    2584 scope.go:117] "RemoveContainer" containerID="aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876"
Sep 15 19:14:56.255318 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:14:56.255298    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876\": container with ID starting with aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876 not found: ID does not exist" containerID="aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876"
Sep 15 19:14:56.255378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.255326    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876"} err="failed to get container status \"aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876\": rpc error: code = NotFound desc = could not find container \"aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876\": container with ID starting with aec907b10e7b6c72f01669e347972e71bd15f7759b3fa728b6d43b3ce1e88876 not found: ID does not exist"
Sep 15 19:14:56.261730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.261699    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"]
Sep 15 19:14:56.264462 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:56.264443    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9de44d/test-model-9de44d-kserve-7789d66fc-qsq55"]
Sep 15 19:14:58.235952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:14:58.235918    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fa3580-f5f8-4ecc-97f7-b491ca162576" path="/var/lib/kubelet/pods/d5fa3580-f5f8-4ecc-97f7-b491ca162576/volumes"
Sep 15 19:15:00.154704 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.154672    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"]
Sep 15 19:15:00.155213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.155189    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fa3580-f5f8-4ecc-97f7-b491ca162576" containerName="main"
Sep 15 19:15:00.155213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.155208    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fa3580-f5f8-4ecc-97f7-b491ca162576" containerName="main"
Sep 15 19:15:00.155360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.155241    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e278d84-55d7-4499-9de1-4eb4c361d678" containerName="revoke-keys"
Sep 15 19:15:00.155360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.155248    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e278d84-55d7-4499-9de1-4eb4c361d678" containerName="revoke-keys"
Sep 15 19:15:00.155360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.155323    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fa3580-f5f8-4ecc-97f7-b491ca162576" containerName="main"
Sep 15 19:15:00.155360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.155330    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e278d84-55d7-4499-9de1-4eb4c361d678" containerName="revoke-keys"
Sep 15 19:15:00.158012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.157991    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"]
Sep 15 19:15:00.158124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.158094    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"
Sep 15 19:15:00.160537 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.160516    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"]
Sep 15 19:15:00.160678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.160524    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:15:00.160678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.160631    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"
Sep 15 19:15:00.162791 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.162770    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"
Sep 15 19:15:00.162927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.162909    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"]
Sep 15 19:15:00.166336 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.166318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"
Sep 15 19:15:00.169753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.169558    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"]
Sep 15 19:15:00.171865 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.171847    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"
Sep 15 19:15:00.174098 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.174081    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"]
Sep 15 19:15:00.177986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.177966    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"]
Sep 15 19:15:00.181970 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.181827    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"
Sep 15 19:15:00.183441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.183421    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"]
Sep 15 19:15:00.187297 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.187277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"
Sep 15 19:15:00.191411 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.191389    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"]
Sep 15 19:15:00.200829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.200806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"]
Sep 15 19:15:00.243686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.243421    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"
Sep 15 19:15:00.255204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.255150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"
Sep 15 19:15:00.296001 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.295972    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"
Sep 15 19:15:00.476434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.476408    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"]
Sep 15 19:15:00.477215 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:15:00.477187    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bf24d4_8d25_4674_afdc_aab1e1707a5e.slice/crio-767756b9d799fdeb69896b5938468034febb101fc1885b78cd41627956bf20eb WatchSource:0}: Error finding container 767756b9d799fdeb69896b5938468034febb101fc1885b78cd41627956bf20eb: Status 404 returned error can't find the container with id 767756b9d799fdeb69896b5938468034febb101fc1885b78cd41627956bf20eb
Sep 15 19:15:00.483253 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.483228    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"]
Sep 15 19:15:00.483560 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:15:00.483540    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfed805_cb6c_40e4_b8e1_f6c04fb91e8b.slice/crio-d7c770e732949ad14e22347df9c89ada699d02575ce50acfd13970c98cbdfd56 WatchSource:0}: Error finding container d7c770e732949ad14e22347df9c89ada699d02575ce50acfd13970c98cbdfd56: Status 404 returned error can't find the container with id d7c770e732949ad14e22347df9c89ada699d02575ce50acfd13970c98cbdfd56
Sep 15 19:15:00.564197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.564169    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"]
Sep 15 19:15:00.564326 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:15:00.564307    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2b8d50_ba38_49ce_b41f_738aa68562ae.slice/crio-24bc5a144c52ffc13fd38f48a7408244c996282d35be5f5ce74324f617cda2ca WatchSource:0}: Error finding container 24bc5a144c52ffc13fd38f48a7408244c996282d35be5f5ce74324f617cda2ca: Status 404 returned error can't find the container with id 24bc5a144c52ffc13fd38f48a7408244c996282d35be5f5ce74324f617cda2ca
Sep 15 19:15:00.587060 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.587036    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"]
Sep 15 19:15:00.592863 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:15:00.592841    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f7751_09ec_4432_a5a2_bf77202976ea.slice/crio-a6e8c288e0d59296cda09e251a8f383f0ea92d5186b9c18f856eda7d51b96562 WatchSource:0}: Error finding container a6e8c288e0d59296cda09e251a8f383f0ea92d5186b9c18f856eda7d51b96562: Status 404 returned error can't find the container with id a6e8c288e0d59296cda09e251a8f383f0ea92d5186b9c18f856eda7d51b96562
Sep 15 19:15:00.596715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:00.596695    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"]
Sep 15 19:15:00.597129 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:15:00.597109    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7effa0_6bcd_4ebf_b478_35c2af84f83e.slice/crio-5a449f90993ad71f228d21e98c35131964943f4e79603d77557c26df89b82ee0 WatchSource:0}: Error finding container 5a449f90993ad71f228d21e98c35131964943f4e79603d77557c26df89b82ee0: Status 404 returned error can't find the container with id 5a449f90993ad71f228d21e98c35131964943f4e79603d77557c26df89b82ee0
Sep 15 19:15:01.308116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.308010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn" event={"ID":"bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b","Type":"ContainerStarted","Data":"c8ffb27125f5a0b7b41c96b1b5b7f900db544a0a29ad1f4f7718b7ab948fe505"}
Sep 15 19:15:01.308116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.308083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn" event={"ID":"bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b","Type":"ContainerStarted","Data":"d7c770e732949ad14e22347df9c89ada699d02575ce50acfd13970c98cbdfd56"}
Sep 15 19:15:01.309748 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.309721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph" event={"ID":"0a7effa0-6bcd-4ebf-b478-35c2af84f83e","Type":"ContainerStarted","Data":"554e7795b127ad285f663ac101508e59a5f84885f618c4c89a452b384f88751b"}
Sep 15 19:15:01.309897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.309766    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph" event={"ID":"0a7effa0-6bcd-4ebf-b478-35c2af84f83e","Type":"ContainerStarted","Data":"5a449f90993ad71f228d21e98c35131964943f4e79603d77557c26df89b82ee0"}
Sep 15 19:15:01.311557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.311525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk" event={"ID":"3d2b8d50-ba38-49ce-b41f-738aa68562ae","Type":"ContainerStarted","Data":"1ff83c140b14b65a18388b86532977e4249a2574b014451bf3c2522a4540632d"}
Sep 15 19:15:01.311727 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.311572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk" event={"ID":"3d2b8d50-ba38-49ce-b41f-738aa68562ae","Type":"ContainerStarted","Data":"24bc5a144c52ffc13fd38f48a7408244c996282d35be5f5ce74324f617cda2ca"}
Sep 15 19:15:01.313301 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.313277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4" event={"ID":"e58f7751-09ec-4432-a5a2-bf77202976ea","Type":"ContainerStarted","Data":"401f3781525c565977280dc19154211d34795f816a0b3617227dc375954c5936"}
Sep 15 19:15:01.313400 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.313324    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4" event={"ID":"e58f7751-09ec-4432-a5a2-bf77202976ea","Type":"ContainerStarted","Data":"a6e8c288e0d59296cda09e251a8f383f0ea92d5186b9c18f856eda7d51b96562"}
Sep 15 19:15:01.315319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.315271    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27" event={"ID":"a3bf24d4-8d25-4674-afdc-aab1e1707a5e","Type":"ContainerStarted","Data":"da4f49f6f78dc6f94406b7f0fada2ba724cd013120a3ac9a44e1b8ca89a4becd"}
Sep 15 19:15:01.315319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.315316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27" event={"ID":"a3bf24d4-8d25-4674-afdc-aab1e1707a5e","Type":"ContainerStarted","Data":"767756b9d799fdeb69896b5938468034febb101fc1885b78cd41627956bf20eb"}
Sep 15 19:15:01.323118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.322107    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn" podStartSLOduration=1.322071979 podStartE2EDuration="1.322071979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:01.320826371 +0000 UTC m=+1453.475194843" watchObservedRunningTime="2026-09-15 19:15:01.322071979 +0000 UTC m=+1453.476440448"
Sep 15 19:15:01.336145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.336102    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27" podStartSLOduration=1.336084133 podStartE2EDuration="1.336084133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:01.334394446 +0000 UTC m=+1453.488762915" watchObservedRunningTime="2026-09-15 19:15:01.336084133 +0000 UTC m=+1453.490452602"
Sep 15 19:15:01.349515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.349475    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph" podStartSLOduration=1.349462163 podStartE2EDuration="1.349462163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:01.346503394 +0000 UTC m=+1453.500871862" watchObservedRunningTime="2026-09-15 19:15:01.349462163 +0000 UTC m=+1453.503830630"
Sep 15 19:15:01.360524 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.360484    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk" podStartSLOduration=1.360470424 podStartE2EDuration="1.360470424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:01.358440112 +0000 UTC m=+1453.512808581" watchObservedRunningTime="2026-09-15 19:15:01.360470424 +0000 UTC m=+1453.514838892"
Sep 15 19:15:01.371309 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:01.371270    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4" podStartSLOduration=1.3712601260000001 podStartE2EDuration="1.371260126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:01.370717564 +0000 UTC m=+1453.525086032" watchObservedRunningTime="2026-09-15 19:15:01.371260126 +0000 UTC m=+1453.525628622"
Sep 15 19:15:02.325473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.325437    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a7effa0-6bcd-4ebf-b478-35c2af84f83e" containerID="554e7795b127ad285f663ac101508e59a5f84885f618c4c89a452b384f88751b" exitCode=0
Sep 15 19:15:02.325928 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.325527    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph" event={"ID":"0a7effa0-6bcd-4ebf-b478-35c2af84f83e","Type":"ContainerDied","Data":"554e7795b127ad285f663ac101508e59a5f84885f618c4c89a452b384f88751b"}
Sep 15 19:15:02.327340 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.327312    2584 generic.go:356] "Generic (PLEG): container finished" podID="3d2b8d50-ba38-49ce-b41f-738aa68562ae" containerID="1ff83c140b14b65a18388b86532977e4249a2574b014451bf3c2522a4540632d" exitCode=0
Sep 15 19:15:02.327448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.327398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk" event={"ID":"3d2b8d50-ba38-49ce-b41f-738aa68562ae","Type":"ContainerDied","Data":"1ff83c140b14b65a18388b86532977e4249a2574b014451bf3c2522a4540632d"}
Sep 15 19:15:02.329123 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.329098    2584 generic.go:356] "Generic (PLEG): container finished" podID="e58f7751-09ec-4432-a5a2-bf77202976ea" containerID="401f3781525c565977280dc19154211d34795f816a0b3617227dc375954c5936" exitCode=0
Sep 15 19:15:02.329250 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.329176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4" event={"ID":"e58f7751-09ec-4432-a5a2-bf77202976ea","Type":"ContainerDied","Data":"401f3781525c565977280dc19154211d34795f816a0b3617227dc375954c5936"}
Sep 15 19:15:02.330803 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.330782    2584 generic.go:356] "Generic (PLEG): container finished" podID="a3bf24d4-8d25-4674-afdc-aab1e1707a5e" containerID="da4f49f6f78dc6f94406b7f0fada2ba724cd013120a3ac9a44e1b8ca89a4becd" exitCode=0
Sep 15 19:15:02.330883 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.330805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27" event={"ID":"a3bf24d4-8d25-4674-afdc-aab1e1707a5e","Type":"ContainerDied","Data":"da4f49f6f78dc6f94406b7f0fada2ba724cd013120a3ac9a44e1b8ca89a4becd"}
Sep 15 19:15:02.332445 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.332424    2584 generic.go:356] "Generic (PLEG): container finished" podID="bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b" containerID="c8ffb27125f5a0b7b41c96b1b5b7f900db544a0a29ad1f4f7718b7ab948fe505" exitCode=0
Sep 15 19:15:02.332514 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:02.332495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn" event={"ID":"bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b","Type":"ContainerDied","Data":"c8ffb27125f5a0b7b41c96b1b5b7f900db544a0a29ad1f4f7718b7ab948fe505"}
Sep 15 19:15:03.496562 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:03.496537    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"
Sep 15 19:15:03.550305 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:03.550279    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"
Sep 15 19:15:03.586076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:03.586056    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"
Sep 15 19:15:03.589570 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:03.589546    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"
Sep 15 19:15:03.593740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:03.593690    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"
Sep 15 19:15:04.348100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.348070    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"
Sep 15 19:15:04.348100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.348089    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn" event={"ID":"bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b","Type":"ContainerDied","Data":"d7c770e732949ad14e22347df9c89ada699d02575ce50acfd13970c98cbdfd56"}
Sep 15 19:15:04.348347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.348161    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c770e732949ad14e22347df9c89ada699d02575ce50acfd13970c98cbdfd56"
Sep 15 19:15:04.349567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.349535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph" event={"ID":"0a7effa0-6bcd-4ebf-b478-35c2af84f83e","Type":"ContainerDied","Data":"5a449f90993ad71f228d21e98c35131964943f4e79603d77557c26df89b82ee0"}
Sep 15 19:15:04.349567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.349550    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"
Sep 15 19:15:04.349797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.349577    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a449f90993ad71f228d21e98c35131964943f4e79603d77557c26df89b82ee0"
Sep 15 19:15:04.351224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.351207    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"
Sep 15 19:15:04.351224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.351214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk" event={"ID":"3d2b8d50-ba38-49ce-b41f-738aa68562ae","Type":"ContainerDied","Data":"24bc5a144c52ffc13fd38f48a7408244c996282d35be5f5ce74324f617cda2ca"}
Sep 15 19:15:04.351400 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.351253    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24bc5a144c52ffc13fd38f48a7408244c996282d35be5f5ce74324f617cda2ca"
Sep 15 19:15:04.352661 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.352638    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4" event={"ID":"e58f7751-09ec-4432-a5a2-bf77202976ea","Type":"ContainerDied","Data":"a6e8c288e0d59296cda09e251a8f383f0ea92d5186b9c18f856eda7d51b96562"}
Sep 15 19:15:04.352785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.352672    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"
Sep 15 19:15:04.352785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.352685    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e8c288e0d59296cda09e251a8f383f0ea92d5186b9c18f856eda7d51b96562"
Sep 15 19:15:04.354565 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.354542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27" event={"ID":"a3bf24d4-8d25-4674-afdc-aab1e1707a5e","Type":"ContainerDied","Data":"767756b9d799fdeb69896b5938468034febb101fc1885b78cd41627956bf20eb"}
Sep 15 19:15:04.354664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.354565    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"
Sep 15 19:15:04.354664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:04.354584    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767756b9d799fdeb69896b5938468034febb101fc1885b78cd41627956bf20eb"
Sep 15 19:15:09.706245 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:09.706214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:16.835725 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:16.835676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:17.557692 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:17.557665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:17.770796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:17.770764    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"]
Sep 15 19:15:17.771018 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:17.770989    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy" containerID="cri-o://21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a" gracePeriod=30
Sep 15 19:15:17.798674 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:17.798637    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:18.097735 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.097700    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"]
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098158    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58f7751-09ec-4432-a5a2-bf77202976ea" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098169    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58f7751-09ec-4432-a5a2-bf77202976ea" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098181    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098187    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098193    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7effa0-6bcd-4ebf-b478-35c2af84f83e" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098198    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7effa0-6bcd-4ebf-b478-35c2af84f83e" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098207    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2b8d50-ba38-49ce-b41f-738aa68562ae" containerName="cleanup"
Sep 15 19:15:18.098209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098212    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2b8d50-ba38-49ce-b41f-738aa68562ae" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098219    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3bf24d4-8d25-4674-afdc-aab1e1707a5e" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098224    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3bf24d4-8d25-4674-afdc-aab1e1707a5e" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098317    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7effa0-6bcd-4ebf-b478-35c2af84f83e" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098323    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3bf24d4-8d25-4674-afdc-aab1e1707a5e" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098331    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2b8d50-ba38-49ce-b41f-738aa68562ae" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098336    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58f7751-09ec-4432-a5a2-bf77202976ea" containerName="cleanup"
Sep 15 19:15:18.098683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.098346    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b" containerName="cleanup"
Sep 15 19:15:18.101848 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.101826    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:18.104488 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.104462    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:15:18.104683 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.104462    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:15:18.108251 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.108229    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"]
Sep 15 19:15:18.163895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.163851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86407b5d-c172-4d6a-9944-1817cddf5c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl\" (UID: \"86407b5d-c172-4d6a-9944-1817cddf5c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:18.233744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.233696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:18.264902 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.264860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86407b5d-c172-4d6a-9944-1817cddf5c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl\" (UID: \"86407b5d-c172-4d6a-9944-1817cddf5c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:18.265496 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.265478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86407b5d-c172-4d6a-9944-1817cddf5c9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl\" (UID: \"86407b5d-c172-4d6a-9944-1817cddf5c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:18.414694 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.414567    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:18.690505 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:18.690479    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"]
Sep 15 19:15:18.692190 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:15:18.692161    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86407b5d_c172_4d6a_9944_1817cddf5c9d.slice/crio-25f2cca51328168e97548d5b8d0f125b84644dfcb4991b12aa14f1f3531139ce WatchSource:0}: Error finding container 25f2cca51328168e97548d5b8d0f125b84644dfcb4991b12aa14f1f3531139ce: Status 404 returned error can't find the container with id 25f2cca51328168e97548d5b8d0f125b84644dfcb4991b12aa14f1f3531139ce
Sep 15 19:15:19.016820 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:19.016791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:19.461327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:19.461242    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl" event={"ID":"86407b5d-c172-4d6a-9944-1817cddf5c9d","Type":"ContainerStarted","Data":"821773835e9daac1136bca68324ab575c0f8b783f2a38512e9b36b4fdc4c3c9f"}
Sep 15 19:15:19.461327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:19.461297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl" event={"ID":"86407b5d-c172-4d6a-9944-1817cddf5c9d","Type":"ContainerStarted","Data":"25f2cca51328168e97548d5b8d0f125b84644dfcb4991b12aa14f1f3531139ce"}
Sep 15 19:15:19.477284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:19.477238    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl" podStartSLOduration=1.477223458 podStartE2EDuration="1.477223458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:19.476275703 +0000 UTC m=+1471.630644171" watchObservedRunningTime="2026-09-15 19:15:19.477223458 +0000 UTC m=+1471.631591926"
Sep 15 19:15:19.547231 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:19.547201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:20.323415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:20.323373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:20.470260 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:20.470223    2584 generic.go:356] "Generic (PLEG): container finished" podID="86407b5d-c172-4d6a-9944-1817cddf5c9d" containerID="821773835e9daac1136bca68324ab575c0f8b783f2a38512e9b36b4fdc4c3c9f" exitCode=0
Sep 15 19:15:20.470724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:20.470306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl" event={"ID":"86407b5d-c172-4d6a-9944-1817cddf5c9d","Type":"ContainerDied","Data":"821773835e9daac1136bca68324ab575c0f8b783f2a38512e9b36b4fdc4c3c9f"}
Sep 15 19:15:20.933559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:20.933520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:21.575844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:21.575781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:21.618725 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:21.618705    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:21.701767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:21.701733    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86407b5d-c172-4d6a-9944-1817cddf5c9d-maas-api-service-ca\") pod \"86407b5d-c172-4d6a-9944-1817cddf5c9d\" (UID: \"86407b5d-c172-4d6a-9944-1817cddf5c9d\") "
Sep 15 19:15:21.702065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:21.702043    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86407b5d-c172-4d6a-9944-1817cddf5c9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86407b5d-c172-4d6a-9944-1817cddf5c9d" (UID: "86407b5d-c172-4d6a-9944-1817cddf5c9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:15:21.702140 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:21.702126    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86407b5d-c172-4d6a-9944-1817cddf5c9d-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:22.317899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:22.317872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:22.485880 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:22.485856    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"
Sep 15 19:15:22.486047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:22.485858    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl" event={"ID":"86407b5d-c172-4d6a-9944-1817cddf5c9d","Type":"ContainerDied","Data":"25f2cca51328168e97548d5b8d0f125b84644dfcb4991b12aa14f1f3531139ce"}
Sep 15 19:15:22.486047 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:22.485983    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f2cca51328168e97548d5b8d0f125b84644dfcb4991b12aa14f1f3531139ce"
Sep 15 19:15:22.948834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:22.948807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:23.045748 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.045684    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:15:23.130732 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.130703    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-istio-token\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.130889 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.130758    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646a529b-e291-4e71-b761-c838288e91e4-istiod-ca-cert\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.130944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.130886    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-credential-socket\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.130944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.130922    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-envoy\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.131065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.130953    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646a529b-e291-4e71-b761-c838288e91e4-istio-podinfo\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.131065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.130993    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-certs\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.131172 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131067    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv2jk\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-kube-api-access-nv2jk\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.131172 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-socket\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.131172 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131129    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-data\") pod \"646a529b-e291-4e71-b761-c838288e91e4\" (UID: \"646a529b-e291-4e71-b761-c838288e91e4\") "
Sep 15 19:15:23.131172 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131158    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:23.131387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131169    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646a529b-e291-4e71-b761-c838288e91e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:15:23.131387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131289    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:23.131481 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131392    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:23.131481 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131425    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:23.131749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131719    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/646a529b-e291-4e71-b761-c838288e91e4-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.131749 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131743    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.131917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131757    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.131917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131770    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.131917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.131784    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.133260 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.133230    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/646a529b-e291-4e71-b761-c838288e91e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:15:23.133384 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.133266    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:23.133384 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.133285    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:23.133496 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.133389    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-kube-api-access-nv2jk" (OuterVolumeSpecName: "kube-api-access-nv2jk") pod "646a529b-e291-4e71-b761-c838288e91e4" (UID: "646a529b-e291-4e71-b761-c838288e91e4"). InnerVolumeSpecName "kube-api-access-nv2jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:23.232877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.232844    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2jk\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-kube-api-access-nv2jk\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.232877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.232873    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/646a529b-e291-4e71-b761-c838288e91e4-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.233043 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.232886    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/646a529b-e291-4e71-b761-c838288e91e4-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.233043 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.232897    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/646a529b-e291-4e71-b761-c838288e91e4-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:23.494210 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.494123    2584 generic.go:356] "Generic (PLEG): container finished" podID="646a529b-e291-4e71-b761-c838288e91e4" containerID="21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a" exitCode=0
Sep 15 19:15:23.494210 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.494177    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"
Sep 15 19:15:23.494450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.494214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" event={"ID":"646a529b-e291-4e71-b761-c838288e91e4","Type":"ContainerDied","Data":"21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a"}
Sep 15 19:15:23.494450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.494279    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz" event={"ID":"646a529b-e291-4e71-b761-c838288e91e4","Type":"ContainerDied","Data":"6dbce975f2bdb0909a995c12054e883344cc4764d1e65d0f1f864cf57e00f60c"}
Sep 15 19:15:23.494450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.494310    2584 scope.go:117] "RemoveContainer" containerID="21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a"
Sep 15 19:15:23.507577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.507557    2584 scope.go:117] "RemoveContainer" containerID="0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"
Sep 15 19:15:23.516896 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.516878    2584 scope.go:117] "RemoveContainer" containerID="21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a"
Sep 15 19:15:23.517162 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:15:23.517142    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a\": container with ID starting with 21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a not found: ID does not exist" containerID="21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a"
Sep 15 19:15:23.517228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.517170    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a"} err="failed to get container status \"21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a\": rpc error: code = NotFound desc = could not find container \"21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a\": container with ID starting with 21074cc2d93919ac27beb8a076430abd51c30588797a03c78736f09fc2e84d5a not found: ID does not exist"
Sep 15 19:15:23.517228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.517186    2584 scope.go:117] "RemoveContainer" containerID="0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"
Sep 15 19:15:23.517450 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:15:23.517433    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b\": container with ID starting with 0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b not found: ID does not exist" containerID="0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"
Sep 15 19:15:23.517499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.517456    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b"} err="failed to get container status \"0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b\": rpc error: code = NotFound desc = could not find container \"0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b\": container with ID starting with 0971ba2dcf0db913eaedca8aaec89e4f1dd07e397aae3f56bfd5dd688f2f771b not found: ID does not exist"
Sep 15 19:15:23.519418 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.519398    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"]
Sep 15 19:15:23.523645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:23.523598    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9de44d-openshift-default-78f95cffcd-vljdz"]
Sep 15 19:15:24.234906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:24.234873    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646a529b-e291-4e71-b761-c838288e91e4" path="/var/lib/kubelet/pods/646a529b-e291-4e71-b761-c838288e91e4/volumes"
Sep 15 19:15:28.646019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.645982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:28.648248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.648228    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"]
Sep 15 19:15:28.651420 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.651396    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daa58-29824995-xw2s4"]
Sep 15 19:15:28.719057 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.719015    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:28.779203 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.779163    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:28.841255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.841224    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"]
Sep 15 19:15:28.843828 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.843800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:28.845440 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:28.845421    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daa58-5g6zl"]
Sep 15 19:15:30.235300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:30.235266    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86407b5d-c172-4d6a-9944-1817cddf5c9d" path="/var/lib/kubelet/pods/86407b5d-c172-4d6a-9944-1817cddf5c9d/volumes"
Sep 15 19:15:30.235815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:30.235723    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e58f7751-09ec-4432-a5a2-bf77202976ea" path="/var/lib/kubelet/pods/e58f7751-09ec-4432-a5a2-bf77202976ea/volumes"
Sep 15 19:15:30.434980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:30.434948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:30.484290 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:30.484263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:30.550282 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:30.550243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:31.312133 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:31.312099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:31.912391 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:31.912357    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:32.803263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:32.803225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:35.004622 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:35.004589    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:35.232551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:35.232522    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"]
Sep 15 19:15:35.232809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:35.232784    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy" containerID="cri-o://0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f" gracePeriod=30
Sep 15 19:15:36.037098 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037066    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"]
Sep 15 19:15:36.037728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037707    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy"
Sep 15 19:15:36.037728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037729    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy"
Sep 15 19:15:36.037886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037755    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86407b5d-c172-4d6a-9944-1817cddf5c9d" containerName="revoke-keys"
Sep 15 19:15:36.037886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037763    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="86407b5d-c172-4d6a-9944-1817cddf5c9d" containerName="revoke-keys"
Sep 15 19:15:36.037985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037897    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="86407b5d-c172-4d6a-9944-1817cddf5c9d" containerName="revoke-keys"
Sep 15 19:15:36.037985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037913    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy"
Sep 15 19:15:36.037985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.037921    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy"
Sep 15 19:15:36.042821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.042799    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.045572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.045552    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c6d10a-openshift-default-dockercfg-9dgrk\""
Sep 15 19:15:36.053453 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.051854    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"]
Sep 15 19:15:36.060499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsmb\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-kube-api-access-btsmb\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060530    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-podinfo\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-certs\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-socket\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060721    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-credential-socket\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060887 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060807    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-envoy\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060887 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060874    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-token\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.060993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060931    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istiod-ca-cert\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.061052 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.060993    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-data\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.162268 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.162223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-podinfo\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.162790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.162767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-certs\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.162964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.162950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-socket\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-credential-socket\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163180 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-envoy\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-token\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163416    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istiod-ca-cert\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-data\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163743 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163719    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-credential-socket\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163833 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsmb\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-kube-api-access-btsmb\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.163893 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.163449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-certs\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.164596 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.164572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-socket\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.170658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.166961    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istiod-ca-cert\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.170658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.167629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-data\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.175625 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.173825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-envoy\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.176155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.176129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-podinfo\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.184813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.184786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsmb\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-kube-api-access-btsmb\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.185341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.185306    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-token\") pod \"e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") " pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.357946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.357802    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:36.640792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.640762    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"]
Sep 15 19:15:36.645044 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.642746    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:15:36.645044 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.642922    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:15:36.645044 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:36.643024    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:15:37.597223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:37.597176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" event={"ID":"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a","Type":"ContainerStarted","Data":"2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"}
Sep 15 19:15:37.597723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:37.597253    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" event={"ID":"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a","Type":"ContainerStarted","Data":"eb42705c3d023f0e56276f694fa1745d66b40fe96815d234f684c2eb3d02b258"}
Sep 15 19:15:37.620790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:37.620729    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podStartSLOduration=1.620707277 podStartE2EDuration="1.620707277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:15:37.616366387 +0000 UTC m=+1489.770734865" watchObservedRunningTime="2026-09-15 19:15:37.620707277 +0000 UTC m=+1489.775075746"
Sep 15 19:15:38.358685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:38.358652    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:38.363530 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:38.363504    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:38.604941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:38.604912    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:38.605722 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:38.605703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:15:39.265492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:39.265461    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:39.506414 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:39.506381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:39.600507 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:39.600438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:40.128656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.128624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:40.489839 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.489816    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:15:40.508423 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508394    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983248b6-b680-4c9e-bb1e-bc22ae769201-istiod-ca-cert\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508440    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-podinfo\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508486    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-socket\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508510    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-envoy\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508534    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-certs\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508762    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z48z9\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-kube-api-access-z48z9\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508854 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508801    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-credential-socket\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.508854 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508803    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:40.508854 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508827    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-data\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.509003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508846    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:40.509003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508858    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983248b6-b680-4c9e-bb1e-bc22ae769201-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:15:40.509003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.508892    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-token\") pod \"983248b6-b680-4c9e-bb1e-bc22ae769201\" (UID: \"983248b6-b680-4c9e-bb1e-bc22ae769201\") "
Sep 15 19:15:40.509153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509091    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:40.509153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509107    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-data" (OuterVolumeSpecName: "istio-data") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:40.509383 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509359    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.509510 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509385    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.509510 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509401    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/983248b6-b680-4c9e-bb1e-bc22ae769201-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.509510 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509412    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.509510 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.509424    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.510794 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.510769    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:15:40.510886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.510838    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:15:40.510946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.510910    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-kube-api-access-z48z9" (OuterVolumeSpecName: "kube-api-access-z48z9") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "kube-api-access-z48z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:40.510946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.510921    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-token" (OuterVolumeSpecName: "istio-token") pod "983248b6-b680-4c9e-bb1e-bc22ae769201" (UID: "983248b6-b680-4c9e-bb1e-bc22ae769201"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:15:40.610788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.610753    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z48z9\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-kube-api-access-z48z9\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.610788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.610783    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.610788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.610793    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.611012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.610802    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/983248b6-b680-4c9e-bb1e-bc22ae769201-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:15:40.621183 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.621146    2584 generic.go:356] "Generic (PLEG): container finished" podID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerID="0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f" exitCode=0
Sep 15 19:15:40.621364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.621223    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"
Sep 15 19:15:40.621364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.621237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" event={"ID":"983248b6-b680-4c9e-bb1e-bc22ae769201","Type":"ContainerDied","Data":"0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f"}
Sep 15 19:15:40.621364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.621338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469" event={"ID":"983248b6-b680-4c9e-bb1e-bc22ae769201","Type":"ContainerDied","Data":"97bc1070f7df347001c8ed9b94fcf5bae960d5e8edde7314dedde360b6507598"}
Sep 15 19:15:40.621532 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.621370    2584 scope.go:117] "RemoveContainer" containerID="0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f"
Sep 15 19:15:40.632539 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.632523    2584 scope.go:117] "RemoveContainer" containerID="701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"
Sep 15 19:15:40.644442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.644420    2584 scope.go:117] "RemoveContainer" containerID="0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f"
Sep 15 19:15:40.644940 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:15:40.644916    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f\": container with ID starting with 0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f not found: ID does not exist" containerID="0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f"
Sep 15 19:15:40.645048 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.644951    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f"} err="failed to get container status \"0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f\": rpc error: code = NotFound desc = could not find container \"0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f\": container with ID starting with 0f3654896542cc2f35e5bf960c3f0b7c3d47892d5bb7a759869f32219954169f not found: ID does not exist"
Sep 15 19:15:40.645048 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.644985    2584 scope.go:117] "RemoveContainer" containerID="701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"
Sep 15 19:15:40.645267 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:15:40.645246    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133\": container with ID starting with 701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133 not found: ID does not exist" containerID="701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"
Sep 15 19:15:40.645329 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.645273    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133"} err="failed to get container status \"701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133\": rpc error: code = NotFound desc = could not find container \"701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133\": container with ID starting with 701ca80decb1e79bed3fdb05b5b3d8763d629d4c8af67d9c4ba43fcd124a7133 not found: ID does not exist"
Sep 15 19:15:40.648558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.648535    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"]
Sep 15 19:15:40.653796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:40.653746    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3daa58-openshift-default-5b7b79f9b-rs469"]
Sep 15 19:15:41.026917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:41.026882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:41.723204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:41.723147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:42.235355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:42.235320    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" path="/var/lib/kubelet/pods/983248b6-b680-4c9e-bb1e-bc22ae769201/volumes"
Sep 15 19:15:42.396311 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:42.396267    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:43.291930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:43.291889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:43.365742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:43.365700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:43.466924 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:43.466895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:43.758303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:43.758275    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:43.840316 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:43.840284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:15:48.506041 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:48.506007    2584 scope.go:117] "RemoveContainer" containerID="884365e120d82916d332ef6f04223123d76af62a275d78682548a2c0a7312423"
Sep 15 19:15:48.515480 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:15:48.515457    2584 scope.go:117] "RemoveContainer" containerID="bdf6305a96d281110071f2abf26709c8611fc9a49b4349ae61cb648bef2e054a"
Sep 15 19:16:04.463762 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.463725    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"]
Sep 15 19:16:04.464368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.464345    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy"
Sep 15 19:16:04.464460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.464370    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="646a529b-e291-4e71-b761-c838288e91e4" containerName="istio-proxy"
Sep 15 19:16:04.464460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.464389    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy"
Sep 15 19:16:04.464460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.464397    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy"
Sep 15 19:16:04.464634 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.464540    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy"
Sep 15 19:16:04.464634 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.464555    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy"
Sep 15 19:16:04.466796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.466776    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.469227 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.469208    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-17c562de\"/\"kube-root-ca.crt\""
Sep 15 19:16:04.470410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.470392    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-17c562de\"/\"default-dockercfg-h2kxp\""
Sep 15 19:16:04.470516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.470442    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-17c562de\"/\"openshift-service-ca.crt\""
Sep 15 19:16:04.470606 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.470592    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-17c562de\"/\"e2e-lifecycle-model-17c562de-kserve-self-signed-certs\""
Sep 15 19:16:04.475142 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.475123    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"]
Sep 15 19:16:04.564513 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.564426    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:04.582605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.582571    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1824e39-53e4-49bd-b3ae-a7843710a539-tls-certs\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.582775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.582644    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-tmp-dir\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.582775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.582744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7xl\" (UniqueName: \"kubernetes.io/projected/a1824e39-53e4-49bd-b3ae-a7843710a539-kube-api-access-bl7xl\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.582775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.582768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-model-cache\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.582947 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.582800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-dshm\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.582947 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.582816    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-home\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7xl\" (UniqueName: \"kubernetes.io/projected/a1824e39-53e4-49bd-b3ae-a7843710a539-kube-api-access-bl7xl\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-model-cache\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684106    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-dshm\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-home\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684409 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1824e39-53e4-49bd-b3ae-a7843710a539-tls-certs\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684409 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-tmp-dir\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684551    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-home\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684659 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-model-cache\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.684741 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.684723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-tmp-dir\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.686322 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.686303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-dshm\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.686585 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.686567    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1824e39-53e4-49bd-b3ae-a7843710a539-tls-certs\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.691474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.691449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7xl\" (UniqueName: \"kubernetes.io/projected/a1824e39-53e4-49bd-b3ae-a7843710a539-kube-api-access-bl7xl\") pod \"e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") " pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.777180 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.777154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:04.814004 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.813264    2584 generic.go:356] "Generic (PLEG): container finished" podID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerID="2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef" exitCode=0
Sep 15 19:16:04.814004 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.813514    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" event={"ID":"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a","Type":"ContainerDied","Data":"2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"}
Sep 15 19:16:04.814557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.814124    2584 scope.go:117] "RemoveContainer" containerID="2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"
Sep 15 19:16:04.827859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.816167    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:04.827859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.816265    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:04.827859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:04.816309    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:05.084293 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.084261    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"]
Sep 15 19:16:05.084709 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:16:05.084683    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1824e39_53e4_49bd_b3ae_a7843710a539.slice/crio-50cfddd605b6018540376a703e9f3f1a9c1c9dde0756f0eb1e8d86c2d8a34a4f WatchSource:0}: Error finding container 50cfddd605b6018540376a703e9f3f1a9c1c9dde0756f0eb1e8d86c2d8a34a4f: Status 404 returned error can't find the container with id 50cfddd605b6018540376a703e9f3f1a9c1c9dde0756f0eb1e8d86c2d8a34a4f
Sep 15 19:16:05.139401 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.139269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:05.718101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.718066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:05.823235 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.823193    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z" event={"ID":"a1824e39-53e4-49bd-b3ae-a7843710a539","Type":"ContainerStarted","Data":"0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef"}
Sep 15 19:16:05.823440 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.823267    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z" event={"ID":"a1824e39-53e4-49bd-b3ae-a7843710a539","Type":"ContainerStarted","Data":"50cfddd605b6018540376a703e9f3f1a9c1c9dde0756f0eb1e8d86c2d8a34a4f"}
Sep 15 19:16:05.827353 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.827315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" event={"ID":"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a","Type":"ContainerStarted","Data":"959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39"}
Sep 15 19:16:05.845051 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:05.844988    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z" podStartSLOduration=1.844965868 podStartE2EDuration="1.844965868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:05.843988055 +0000 UTC m=+1517.998356524" watchObservedRunningTime="2026-09-15 19:16:05.844965868 +0000 UTC m=+1517.999334337"
Sep 15 19:16:06.348314 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:06.348278    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:06.358848 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:06.358817    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:16:06.360072 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:06.360047    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 15 19:16:06.360166 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:06.360095    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 15 19:16:07.211535 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:07.211498    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:07.358900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:07.358867    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 15 19:16:07.359078 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:07.358939    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 15 19:16:07.793080 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:07.793043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:08.358419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:08.358385    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 15 19:16:08.358886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:08.358445    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 15 19:16:08.476512 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:08.476480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:08.605281 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:08.605239    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:16:09.326011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:09.325980    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:09.358912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:09.358878    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 15 19:16:09.359303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:09.358936    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 15 19:16:09.632964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:09.632823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:10.300455 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.300422    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"]
Sep 15 19:16:10.301158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.301140    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy"
Sep 15 19:16:10.301158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.301160    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="983248b6-b680-4c9e-bb1e-bc22ae769201" containerName="istio-proxy"
Sep 15 19:16:10.304385 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.304365    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.308527 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.308509    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-415f87-openshift-default-dockercfg-q9mxs\""
Sep 15 19:16:10.318931 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.318907    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"]
Sep 15 19:16:10.355429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.355381    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-credential-socket\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.355759 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.355730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l62sr\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-kube-api-access-l62sr\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.355992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.355970    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-podinfo\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.356125 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.356108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-envoy\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.356288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.356273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-token\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.356407 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.356392    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-socket\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.356521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.356507    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8feba5e-d19d-458a-83dd-c37683f36e75-istiod-ca-cert\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.356660 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.356645    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-certs\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.356828 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.356810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-data\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.359171 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.359151    2584 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 15 19:16:10.359513 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.359493    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 15 19:16:10.461327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.461290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-podinfo\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.462124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.462095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-envoy\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.462724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.462703    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-token\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.463217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.463199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-socket\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.463355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.463338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8feba5e-d19d-458a-83dd-c37683f36e75-istiod-ca-cert\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.463574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.463556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-certs\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.464470 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.464451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-data\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.464686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.464666    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-credential-socket\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.465077 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.465057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-data\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.465192 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.463967    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-socket\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.465285 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.464201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-certs\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.465460 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.464370    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8feba5e-d19d-458a-83dd-c37683f36e75-istiod-ca-cert\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.465670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.465651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-envoy\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.467152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.465964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-podinfo\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.467152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.466244    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-credential-socket\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.467152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.466333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l62sr\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-kube-api-access-l62sr\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.471250 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.471223    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-token\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.474158 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.474138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l62sr\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-kube-api-access-l62sr\") pod \"e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") " pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.615732 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.615653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:10.925101 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:16:10.925039    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8feba5e_d19d_458a_83dd_c37683f36e75.slice/crio-faf7930814493c0b77e2ec2a17bcd2ccd52475cd22bdaa8206108e78cb6bc9b5 WatchSource:0}: Error finding container faf7930814493c0b77e2ec2a17bcd2ccd52475cd22bdaa8206108e78cb6bc9b5: Status 404 returned error can't find the container with id faf7930814493c0b77e2ec2a17bcd2ccd52475cd22bdaa8206108e78cb6bc9b5
Sep 15 19:16:10.925241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.925213    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"]
Sep 15 19:16:10.927924 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.927884    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:10.928040 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.927972    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:10.928040 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:10.927999    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:11.059053 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.059016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:11.159421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.159388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:11.362714 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.362685    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:16:11.363834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.363806    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:16:11.895981 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.895942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" event={"ID":"d8feba5e-d19d-458a-83dd-c37683f36e75","Type":"ContainerStarted","Data":"e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"}
Sep 15 19:16:11.896172 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.896007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" event={"ID":"d8feba5e-d19d-458a-83dd-c37683f36e75","Type":"ContainerStarted","Data":"faf7930814493c0b77e2ec2a17bcd2ccd52475cd22bdaa8206108e78cb6bc9b5"}
Sep 15 19:16:11.934342 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:11.934293    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podStartSLOduration=1.934278945 podStartE2EDuration="1.934278945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:16:11.929522311 +0000 UTC m=+1524.083890779" watchObservedRunningTime="2026-09-15 19:16:11.934278945 +0000 UTC m=+1524.088647475"
Sep 15 19:16:12.615864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:12.615819    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:12.621137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:12.621107    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:12.903545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:12.903465    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:12.904339 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:12.904323    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:13.722033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:13.721999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:13.830125 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:13.830091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:13.922199 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:13.922168    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:14.711886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:14.711852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:14.778270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:14.778240    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:14.778838 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:14.778812    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:14.794752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:14.794726    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:14.933526 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:14.933495    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:16:15.466965 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:15.466936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:16.127450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:16.127422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:16.823208 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:16.823173    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:17.471093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:17.471031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:18.234248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:18.234207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:18.330065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:18.330032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:18.523148 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:18.523117    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:18.770953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:18.770924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:18.870600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:18.870513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:40.222954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:40.222917    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:41.039507 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:41.039470    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:41.834890 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:41.834857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:42.117517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.117485    2584 generic.go:356] "Generic (PLEG): container finished" podID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerID="e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd" exitCode=0
Sep 15 19:16:42.117712 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.117562    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" event={"ID":"d8feba5e-d19d-458a-83dd-c37683f36e75","Type":"ContainerDied","Data":"e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"}
Sep 15 19:16:42.117997 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.117983    2584 scope.go:117] "RemoveContainer" containerID="e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"
Sep 15 19:16:42.119226 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.119196    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:42.119280 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.119266    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:42.119319 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.119298    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:16:42.598359 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.598327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:42.903482 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:42.903401    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:43.132041 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.132000    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" event={"ID":"d8feba5e-d19d-458a-83dd-c37683f36e75","Type":"ContainerStarted","Data":"469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9"}
Sep 15 19:16:43.418125 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.418089    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:43.616071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.616035    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:43.617154 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.617131    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 15 19:16:43.617228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.617185    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 15 19:16:43.830279 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.830238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:43.946216 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:43.946170    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:44.616148 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:44.616114    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 15 19:16:44.616331 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:44.616170    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 15 19:16:44.751159 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:44.751117    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:45.382939 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:45.382897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:16:45.616835 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:45.616797    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 15 19:16:45.617009 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:45.616862    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 15 19:16:46.616816 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:46.616776    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 15 19:16:46.617214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:46.616845    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 15 19:16:47.616919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:47.616880    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 15 19:16:47.617289 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:47.616943    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 15 19:16:48.562673 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:48.562638    2584 scope.go:117] "RemoveContainer" containerID="e372cf95c46f56cdf88c1886cd269b10acc7b734eb1587b66759aa850e233c04"
Sep 15 19:16:48.616587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:48.616556    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 15 19:16:48.616745 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:48.616639    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 15 19:16:49.621260 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:49.621221    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:49.621721 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:49.621643    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:16:49.622370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:16:49.622348    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:17:08.298451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:08.298420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:28.195083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.195048    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"]
Sep 15 19:17:28.197667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.197650    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:28.200344 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.200317    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:17:28.200450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.200403    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:17:28.207206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.207185    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"]
Sep 15 19:17:28.340380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.340348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac90fd-c8a3-41a3-b87b-b07b7763fe91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-17c562de-6czvc\" (UID: \"00ac90fd-c8a3-41a3-b87b-b07b7763fe91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:28.402691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.402663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:28.442056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.442020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac90fd-c8a3-41a3-b87b-b07b7763fe91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-17c562de-6czvc\" (UID: \"00ac90fd-c8a3-41a3-b87b-b07b7763fe91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:28.442680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.442662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac90fd-c8a3-41a3-b87b-b07b7763fe91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-17c562de-6czvc\" (UID: \"00ac90fd-c8a3-41a3-b87b-b07b7763fe91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:28.509944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.509912    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:28.783075 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:28.783044    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"]
Sep 15 19:17:28.783914 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:17:28.783891    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ac90fd_c8a3_41a3_b87b_b07b7763fe91.slice/crio-f234f0b64e8cb20f4fe6b3215aada0c0c012962a5385a0d50e66fd53909c599c WatchSource:0}: Error finding container f234f0b64e8cb20f4fe6b3215aada0c0c012962a5385a0d50e66fd53909c599c: Status 404 returned error can't find the container with id f234f0b64e8cb20f4fe6b3215aada0c0c012962a5385a0d50e66fd53909c599c
Sep 15 19:17:29.201237 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:29.201156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:29.467054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:29.466973    2584 generic.go:356] "Generic (PLEG): container finished" podID="00ac90fd-c8a3-41a3-b87b-b07b7763fe91" containerID="5f9ec48625f2969b62d6b72409a9ea2ec9740f87857471e7433a3f2fd827eb34" exitCode=0
Sep 15 19:17:29.467054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:29.467027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc" event={"ID":"00ac90fd-c8a3-41a3-b87b-b07b7763fe91","Type":"ContainerDied","Data":"5f9ec48625f2969b62d6b72409a9ea2ec9740f87857471e7433a3f2fd827eb34"}
Sep 15 19:17:29.467233 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:29.467078    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc" event={"ID":"00ac90fd-c8a3-41a3-b87b-b07b7763fe91","Type":"ContainerStarted","Data":"f234f0b64e8cb20f4fe6b3215aada0c0c012962a5385a0d50e66fd53909c599c"}
Sep 15 19:17:29.932792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:29.932733    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:30.588880 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:30.588850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:30.626798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:30.626768    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:30.768485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:30.768447    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac90fd-c8a3-41a3-b87b-b07b7763fe91-maas-api-service-ca\") pod \"00ac90fd-c8a3-41a3-b87b-b07b7763fe91\" (UID: \"00ac90fd-c8a3-41a3-b87b-b07b7763fe91\") "
Sep 15 19:17:30.768840 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:30.768818    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ac90fd-c8a3-41a3-b87b-b07b7763fe91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00ac90fd-c8a3-41a3-b87b-b07b7763fe91" (UID: "00ac90fd-c8a3-41a3-b87b-b07b7763fe91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:30.869919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:30.869873    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ac90fd-c8a3-41a3-b87b-b07b7763fe91-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:31.409080 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:31.409048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:31.487504 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:31.487474    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"
Sep 15 19:17:31.487504 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:31.487496    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc" event={"ID":"00ac90fd-c8a3-41a3-b87b-b07b7763fe91","Type":"ContainerDied","Data":"f234f0b64e8cb20f4fe6b3215aada0c0c012962a5385a0d50e66fd53909c599c"}
Sep 15 19:17:31.487755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:31.487562    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f234f0b64e8cb20f4fe6b3215aada0c0c012962a5385a0d50e66fd53909c599c"
Sep 15 19:17:32.047071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:32.047031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:32.681025 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:32.680995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:33.283862 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.283828    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"]
Sep 15 19:17:33.284291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.284273    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00ac90fd-c8a3-41a3-b87b-b07b7763fe91" containerName="revoke-keys"
Sep 15 19:17:33.284291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.284284    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="00ac90fd-c8a3-41a3-b87b-b07b7763fe91" containerName="revoke-keys"
Sep 15 19:17:33.284403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.284368    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="00ac90fd-c8a3-41a3-b87b-b07b7763fe91" containerName="revoke-keys"
Sep 15 19:17:33.286564 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.286546    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.290136 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.290112    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"default-dockercfg-hc5bs\""
Sep 15 19:17:33.290255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.290172    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"auth-test-model-c6d10a-kserve-self-signed-certs\""
Sep 15 19:17:33.290255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.290182    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"kube-root-ca.crt\""
Sep 15 19:17:33.290255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.290231    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"openshift-service-ca.crt\""
Sep 15 19:17:33.298397 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.298374    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"]
Sep 15 19:17:33.302743 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.302720    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kln5g\" (UniqueName: \"kubernetes.io/projected/8809b592-d109-4781-9224-26be35de469d-kube-api-access-kln5g\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.302843 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.302787    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-model-cache\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.302922 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.302909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-dshm\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.302967 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.302930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-tmp-dir\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.303013 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.302986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8809b592-d109-4781-9224-26be35de469d-tls-certs\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.303069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.303029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-home\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.308167 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.308148    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:33.404201 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kln5g\" (UniqueName: \"kubernetes.io/projected/8809b592-d109-4781-9224-26be35de469d-kube-api-access-kln5g\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-model-cache\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-dshm\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-tmp-dir\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8809b592-d109-4781-9224-26be35de469d-tls-certs\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404655 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404459    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-home\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404746 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-model-cache\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404810 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-tmp-dir\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.404912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.404893    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-home\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.406679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.406659    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-dshm\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.406960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.406940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8809b592-d109-4781-9224-26be35de469d-tls-certs\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.411992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.411971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kln5g\" (UniqueName: \"kubernetes.io/projected/8809b592-d109-4781-9224-26be35de469d-kube-api-access-kln5g\") pod \"auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.468857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.468821    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:33.596527 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.596450    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:33.875992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:33.875962    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"]
Sep 15 19:17:33.876682 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:17:33.876659    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8809b592_d109_4781_9224_26be35de469d.slice/crio-274d991046a757916bf44953e5651e32d593ca0109806a10b0982137b9ac2eb9 WatchSource:0}: Error finding container 274d991046a757916bf44953e5651e32d593ca0109806a10b0982137b9ac2eb9: Status 404 returned error can't find the container with id 274d991046a757916bf44953e5651e32d593ca0109806a10b0982137b9ac2eb9
Sep 15 19:17:34.077492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:34.077433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:34.515405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:34.515374    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q" event={"ID":"8809b592-d109-4781-9224-26be35de469d","Type":"ContainerStarted","Data":"4f287b9f7140301bfca2174b477fe8acc1075d2ae871251c2eec96909a9abaf6"}
Sep 15 19:17:34.515894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:34.515418    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q" event={"ID":"8809b592-d109-4781-9224-26be35de469d","Type":"ContainerStarted","Data":"274d991046a757916bf44953e5651e32d593ca0109806a10b0982137b9ac2eb9"}
Sep 15 19:17:34.537236 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:34.537180    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q" podStartSLOduration=1.537162049 podStartE2EDuration="1.537162049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:17:34.53439277 +0000 UTC m=+1606.688761237" watchObservedRunningTime="2026-09-15 19:17:34.537162049 +0000 UTC m=+1606.691530516"
Sep 15 19:17:38.505586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.505553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:38.617829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.617792    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"]
Sep 15 19:17:38.621755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.621728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:38.624565 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.624544    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:17:38.626035 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.626012    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:17:38.628345 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.628320    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"]
Sep 15 19:17:38.664860 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.664818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f7720-a22e-474e-9a46-d8f229165c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6\" (UID: \"7c3f7720-a22e-474e-9a46-d8f229165c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:38.766318 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.766233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f7720-a22e-474e-9a46-d8f229165c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6\" (UID: \"7c3f7720-a22e-474e-9a46-d8f229165c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:38.766922 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.766902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f7720-a22e-474e-9a46-d8f229165c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6\" (UID: \"7c3f7720-a22e-474e-9a46-d8f229165c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:38.936823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:38.936789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:39.028953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:39.028830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:39.210203 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:39.210171    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"]
Sep 15 19:17:39.210887 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:17:39.210862    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f7720_a22e_474e_9a46_d8f229165c3e.slice/crio-b7e41b660c741c6ab0ade216b0e7f73ce5b06aa4839db9ceb747799b7c7accc0 WatchSource:0}: Error finding container b7e41b660c741c6ab0ade216b0e7f73ce5b06aa4839db9ceb747799b7c7accc0: Status 404 returned error can't find the container with id b7e41b660c741c6ab0ade216b0e7f73ce5b06aa4839db9ceb747799b7c7accc0
Sep 15 19:17:39.553872 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:39.553834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6" event={"ID":"7c3f7720-a22e-474e-9a46-d8f229165c3e","Type":"ContainerStarted","Data":"b7e41b660c741c6ab0ade216b0e7f73ce5b06aa4839db9ceb747799b7c7accc0"}
Sep 15 19:17:39.846925 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:39.846847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:40.472449 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:40.472417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:40.561779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:40.561746    2584 generic.go:356] "Generic (PLEG): container finished" podID="7c3f7720-a22e-474e-9a46-d8f229165c3e" containerID="f051055f4c44f69d6d362d9615aad2494e6358a979360c21c910ba663bea91ee" exitCode=0
Sep 15 19:17:40.562168 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:40.561831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6" event={"ID":"7c3f7720-a22e-474e-9a46-d8f229165c3e","Type":"ContainerDied","Data":"f051055f4c44f69d6d362d9615aad2494e6358a979360c21c910ba663bea91ee"}
Sep 15 19:17:41.300727 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:41.300656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:41.713904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:41.713880    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:41.799491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:41.799456    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f7720-a22e-474e-9a46-d8f229165c3e-maas-api-service-ca\") pod \"7c3f7720-a22e-474e-9a46-d8f229165c3e\" (UID: \"7c3f7720-a22e-474e-9a46-d8f229165c3e\") "
Sep 15 19:17:41.799880 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:41.799859    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3f7720-a22e-474e-9a46-d8f229165c3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c3f7720-a22e-474e-9a46-d8f229165c3e" (UID: "7c3f7720-a22e-474e-9a46-d8f229165c3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:41.900607 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:41.900528    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3f7720-a22e-474e-9a46-d8f229165c3e-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:41.995023 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:41.994994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:42.577197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:42.577167    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"
Sep 15 19:17:42.577197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:42.577182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6" event={"ID":"7c3f7720-a22e-474e-9a46-d8f229165c3e","Type":"ContainerDied","Data":"b7e41b660c741c6ab0ade216b0e7f73ce5b06aa4839db9ceb747799b7c7accc0"}
Sep 15 19:17:42.577414 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:42.577225    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e41b660c741c6ab0ade216b0e7f73ce5b06aa4839db9ceb747799b7c7accc0"
Sep 15 19:17:42.639813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:42.639780    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:43.324350 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:43.324317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:43.487790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:43.487751    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"]
Sep 15 19:17:43.561963 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:43.561928    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:43.597022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:43.596937    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:43.597022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:43.597012    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:43.609371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:43.609346    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:44.022963 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:44.022931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:44.602542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:44.602511    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:17:49.228713 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.228682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:49.236597 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.236570    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"]
Sep 15 19:17:49.239907 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.239884    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-17c562de-29824995-pr6kn"]
Sep 15 19:17:49.327843 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.327816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:49.429592 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.429564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:49.444387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.444356    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"]
Sep 15 19:17:49.447411 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.447391    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-17c562de-6czvc"]
Sep 15 19:17:49.536715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.536683    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:49.932888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:49.932811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:50.234360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.234283    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00ac90fd-c8a3-41a3-b87b-b07b7763fe91" path="/var/lib/kubelet/pods/00ac90fd-c8a3-41a3-b87b-b07b7763fe91/volumes"
Sep 15 19:17:50.234734 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.234602    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b" path="/var/lib/kubelet/pods/bcfed805-cb6c-40e4-b8e1-f6c04fb91e8b/volumes"
Sep 15 19:17:50.473719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.473689    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"]
Sep 15 19:17:50.476659 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.476638    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f0ad35-29824995-cwdkk"]
Sep 15 19:17:50.478452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.478429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:50.498948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.498874    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"]
Sep 15 19:17:50.499155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.499129    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn" podUID="ef5889c3-c6bf-4403-a52d-35c497b0ebe9" containerName="payload-processing" containerID="cri-o://cf58bb312293b68e3037dba28a4b54f652da7ea863ab35550324b969c80a31d5" gracePeriod=30
Sep 15 19:17:50.548423 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.548389    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:50.634163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.634127    2584 generic.go:356] "Generic (PLEG): container finished" podID="ef5889c3-c6bf-4403-a52d-35c497b0ebe9" containerID="cf58bb312293b68e3037dba28a4b54f652da7ea863ab35550324b969c80a31d5" exitCode=0
Sep 15 19:17:50.634345 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.634201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn" event={"ID":"ef5889c3-c6bf-4403-a52d-35c497b0ebe9","Type":"ContainerDied","Data":"cf58bb312293b68e3037dba28a4b54f652da7ea863ab35550324b969c80a31d5"}
Sep 15 19:17:50.647661 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.647622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:50.676167 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.676139    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"]
Sep 15 19:17:50.681243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.681212    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f0ad35-2m8p6"]
Sep 15 19:17:50.747000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.746962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:50.778679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.778655    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:17:50.848037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.847922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:50.911182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.911142    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-plugins-config-volume\") pod \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") "
Sep 15 19:17:50.911367 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.911222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p7bm\" (UniqueName: \"kubernetes.io/projected/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-kube-api-access-4p7bm\") pod \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\" (UID: \"ef5889c3-c6bf-4403-a52d-35c497b0ebe9\") "
Sep 15 19:17:50.911549 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.911523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef5889c3-c6bf-4403-a52d-35c497b0ebe9" (UID: "ef5889c3-c6bf-4403-a52d-35c497b0ebe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:17:50.911709 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.911692    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:50.913394 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:50.913369    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-kube-api-access-4p7bm" (OuterVolumeSpecName: "kube-api-access-4p7bm") pod "ef5889c3-c6bf-4403-a52d-35c497b0ebe9" (UID: "ef5889c3-c6bf-4403-a52d-35c497b0ebe9"). InnerVolumeSpecName "kube-api-access-4p7bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:51.013117 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.013022    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7bm\" (UniqueName: \"kubernetes.io/projected/ef5889c3-c6bf-4403-a52d-35c497b0ebe9-kube-api-access-4p7bm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:51.019499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.019471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:51.451553 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.451419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:51.642862 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.642824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn" event={"ID":"ef5889c3-c6bf-4403-a52d-35c497b0ebe9","Type":"ContainerDied","Data":"38fd135b8bd219bdf537f00dab7662861261572b80bec1072a6f0b33078610ed"}
Sep 15 19:17:51.643056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.642893    2584 scope.go:117] "RemoveContainer" containerID="cf58bb312293b68e3037dba28a4b54f652da7ea863ab35550324b969c80a31d5"
Sep 15 19:17:51.643056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.642839    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"
Sep 15 19:17:51.665303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.665254    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"]
Sep 15 19:17:51.668914 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:51.668884    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f0ad35-654f658459-9zvwn"]
Sep 15 19:17:52.234899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:52.234864    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2b8d50-ba38-49ce-b41f-738aa68562ae" path="/var/lib/kubelet/pods/3d2b8d50-ba38-49ce-b41f-738aa68562ae/volumes"
Sep 15 19:17:52.235211 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:52.235197    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3f7720-a22e-474e-9a46-d8f229165c3e" path="/var/lib/kubelet/pods/7c3f7720-a22e-474e-9a46-d8f229165c3e/volumes"
Sep 15 19:17:52.235554 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:52.235542    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5889c3-c6bf-4403-a52d-35c497b0ebe9" path="/var/lib/kubelet/pods/ef5889c3-c6bf-4403-a52d-35c497b0ebe9/volumes"
Sep 15 19:17:52.422218 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:52.422164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:52.520867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:52.520835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:53.047998 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:53.047948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:53.690856 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:53.690808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:54.441434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:54.441376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:55.019349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:55.019320    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:56.729480 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:56.729443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:56.828988 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:56.828930    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:56.965866 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:56.965831    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"]
Sep 15 19:17:56.966111 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:56.966088    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" containerID="cri-o://f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a" gracePeriod=30
Sep 15 19:17:57.456374 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:57.456339    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"]
Sep 15 19:17:57.739773 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:57.739675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:57.840782 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:57.840751    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:17:58.507441 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.507406    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z" podUID="3d83151e-d37c-42e9-b9f4-17411dbe0ca3" containerName="main" containerID="cri-o://74c738aec27eb0fed55d6ff308f8adb9edf06e2d72816a64c39518775eb75177" gracePeriod=45
Sep 15 19:17:58.712474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.712440    2584 generic.go:356] "Generic (PLEG): container finished" podID="3d83151e-d37c-42e9-b9f4-17411dbe0ca3" containerID="74c738aec27eb0fed55d6ff308f8adb9edf06e2d72816a64c39518775eb75177" exitCode=0
Sep 15 19:17:58.712668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.712523    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z" event={"ID":"3d83151e-d37c-42e9-b9f4-17411dbe0ca3","Type":"ContainerDied","Data":"74c738aec27eb0fed55d6ff308f8adb9edf06e2d72816a64c39518775eb75177"}
Sep 15 19:17:58.772237 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.772175    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:17:58.848122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.848089    2584 patch_prober.go:28] interesting pod/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:17:58.848284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.848163    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:17:58.909993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.909961    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-model-cache\") pod \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") "
Sep 15 19:17:58.910187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910041    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tls-certs\") pod \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") "
Sep 15 19:17:58.910187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910068    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-home\") pod \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") "
Sep 15 19:17:58.910187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910092    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tmp-dir\") pod \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") "
Sep 15 19:17:58.910187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910144    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-dshm\") pod \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") "
Sep 15 19:17:58.910398 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910203    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb2kq\" (UniqueName: \"kubernetes.io/projected/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-kube-api-access-pb2kq\") pod \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\" (UID: \"3d83151e-d37c-42e9-b9f4-17411dbe0ca3\") "
Sep 15 19:17:58.910398 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910237    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d83151e-d37c-42e9-b9f4-17411dbe0ca3" (UID: "3d83151e-d37c-42e9-b9f4-17411dbe0ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.910498 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910443    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d83151e-d37c-42e9-b9f4-17411dbe0ca3" (UID: "3d83151e-d37c-42e9-b9f4-17411dbe0ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.910760 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910738    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.910863 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910762    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:58.910973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.910948    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-home" (OuterVolumeSpecName: "home") pod "3d83151e-d37c-42e9-b9f4-17411dbe0ca3" (UID: "3d83151e-d37c-42e9-b9f4-17411dbe0ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.912276 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.912244    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-dshm" (OuterVolumeSpecName: "dshm") pod "3d83151e-d37c-42e9-b9f4-17411dbe0ca3" (UID: "3d83151e-d37c-42e9-b9f4-17411dbe0ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:17:58.912382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.912324    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d83151e-d37c-42e9-b9f4-17411dbe0ca3" (UID: "3d83151e-d37c-42e9-b9f4-17411dbe0ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:17:58.912382 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:58.912354    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-kube-api-access-pb2kq" (OuterVolumeSpecName: "kube-api-access-pb2kq") pod "3d83151e-d37c-42e9-b9f4-17411dbe0ca3" (UID: "3d83151e-d37c-42e9-b9f4-17411dbe0ca3"). InnerVolumeSpecName "kube-api-access-pb2kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:17:59.012101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.012066    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.012101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.012094    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.012101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.012103    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.012336 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.012113    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2kq\" (UniqueName: \"kubernetes.io/projected/3d83151e-d37c-42e9-b9f4-17411dbe0ca3-kube-api-access-pb2kq\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:17:59.721402 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.721370    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"
Sep 15 19:17:59.721600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.721401    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z" event={"ID":"3d83151e-d37c-42e9-b9f4-17411dbe0ca3","Type":"ContainerDied","Data":"7eabb0dbb8cdfa640dd45012555c7ba8050de58b45b89069dd33e37aec3f82d2"}
Sep 15 19:17:59.721600 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.721461    2584 scope.go:117] "RemoveContainer" containerID="74c738aec27eb0fed55d6ff308f8adb9edf06e2d72816a64c39518775eb75177"
Sep 15 19:17:59.748071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.748045    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"]
Sep 15 19:17:59.751799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:17:59.751777    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f0ad35/distinct-2-w3-f0ad35-kserve-5687844cb-rpc6z"]
Sep 15 19:18:00.235370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:00.235338    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d83151e-d37c-42e9-b9f4-17411dbe0ca3" path="/var/lib/kubelet/pods/3d83151e-d37c-42e9-b9f4-17411dbe0ca3/volumes"
Sep 15 19:18:00.559497 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:00.559468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:00.660781 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:00.660755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:02.225017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.224991    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:18:02.245867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.245795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-data\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.245867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.245853    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-envoy\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.245915    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-certs\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.245960    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-credential-socket\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246039    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istiod-ca-cert\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246074    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkt46\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-kube-api-access-lkt46\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246137    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-podinfo\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246196    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-socket\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246221    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-token\") pod \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\" (UID: \"5e57c12e-68ef-4451-9be0-7a4a9576c8d0\") "
Sep 15 19:18:02.246363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246245    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:02.246472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246363    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:02.246533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:02.246597 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246539    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:02.246802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246750    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.246888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246807    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.246888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246819    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.246888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246830    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.247054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.246915    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:18:02.248780 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.248744    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:18:02.248780 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.248765    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-kube-api-access-lkt46" (OuterVolumeSpecName: "kube-api-access-lkt46") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "kube-api-access-lkt46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:02.248953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.248887    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:02.248953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.248892    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e57c12e-68ef-4451-9be0-7a4a9576c8d0" (UID: "5e57c12e-68ef-4451-9be0-7a4a9576c8d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:02.348215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.348182    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.348215 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.348210    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkt46\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-kube-api-access-lkt46\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.348405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.348226    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.348405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.348239    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.348405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.348251    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e57c12e-68ef-4451-9be0-7a4a9576c8d0-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:02.747930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.747894    2584 generic.go:356] "Generic (PLEG): container finished" podID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerID="f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a" exitCode=0
Sep 15 19:18:02.748106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.747946    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"
Sep 15 19:18:02.748106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.747976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" event={"ID":"5e57c12e-68ef-4451-9be0-7a4a9576c8d0","Type":"ContainerDied","Data":"f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a"}
Sep 15 19:18:02.748106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.748029    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst" event={"ID":"5e57c12e-68ef-4451-9be0-7a4a9576c8d0","Type":"ContainerDied","Data":"ac2a6dca6e9ddf944a604589deb5bfebae3020aeb4cc6b11458d1a95fd825108"}
Sep 15 19:18:02.748106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.748053    2584 scope.go:117] "RemoveContainer" containerID="f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a"
Sep 15 19:18:02.758724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.758679    2584 scope.go:117] "RemoveContainer" containerID="47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"
Sep 15 19:18:02.768169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.768139    2584 scope.go:117] "RemoveContainer" containerID="f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a"
Sep 15 19:18:02.768476 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:18:02.768455    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a\": container with ID starting with f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a not found: ID does not exist" containerID="f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a"
Sep 15 19:18:02.768572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.768490    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a"} err="failed to get container status \"f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a\": rpc error: code = NotFound desc = could not find container \"f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a\": container with ID starting with f047cf2a807896c2526fb1fbcea123ac2fb87d3d202166812c3e2f6c2b284e7a not found: ID does not exist"
Sep 15 19:18:02.768572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.768514    2584 scope.go:117] "RemoveContainer" containerID="47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"
Sep 15 19:18:02.768829 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:18:02.768811    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae\": container with ID starting with 47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae not found: ID does not exist" containerID="47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"
Sep 15 19:18:02.768879 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.768836    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae"} err="failed to get container status \"47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae\": rpc error: code = NotFound desc = could not find container \"47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae\": container with ID starting with 47eae45fbaa6d4f1fd715188712c5494b632f95133483bbe640c8298e371bdae not found: ID does not exist"
Sep 15 19:18:02.770605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.770582    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"]
Sep 15 19:18:02.773957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:02.773936    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f0ad35-openshift-default-5d5478f4cb-d7jst"]
Sep 15 19:18:04.234763 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:04.234733    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" path="/var/lib/kubelet/pods/5e57c12e-68ef-4451-9be0-7a4a9576c8d0/volumes"
Sep 15 19:18:11.502050 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:11.502021    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:12.476602 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.476563    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z" podUID="a1824e39-53e4-49bd-b3ae-a7843710a539" containerName="main" containerID="cri-o://0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef" gracePeriod=45
Sep 15 19:18:12.748944 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.748918    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:18:12.818571 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.818537    2584 generic.go:356] "Generic (PLEG): container finished" podID="a1824e39-53e4-49bd-b3ae-a7843710a539" containerID="0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef" exitCode=0
Sep 15 19:18:12.818785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.818589    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"
Sep 15 19:18:12.818785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.818650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z" event={"ID":"a1824e39-53e4-49bd-b3ae-a7843710a539","Type":"ContainerDied","Data":"0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef"}
Sep 15 19:18:12.818785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.818699    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z" event={"ID":"a1824e39-53e4-49bd-b3ae-a7843710a539","Type":"ContainerDied","Data":"50cfddd605b6018540376a703e9f3f1a9c1c9dde0756f0eb1e8d86c2d8a34a4f"}
Sep 15 19:18:12.818785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.818720    2584 scope.go:117] "RemoveContainer" containerID="0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef"
Sep 15 19:18:12.830863 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.830842    2584 scope.go:117] "RemoveContainer" containerID="0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef"
Sep 15 19:18:12.831194 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:18:12.831172    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef\": container with ID starting with 0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef not found: ID does not exist" containerID="0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef"
Sep 15 19:18:12.831268 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.831203    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef"} err="failed to get container status \"0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef\": rpc error: code = NotFound desc = could not find container \"0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef\": container with ID starting with 0507549e5e73436af255dea9f00cefd6c1a7d4c8d82e8fc38b29dbad240a3aef not found: ID does not exist"
Sep 15 19:18:12.859681 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.859602    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-dshm\") pod \"a1824e39-53e4-49bd-b3ae-a7843710a539\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") "
Sep 15 19:18:12.859857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.859719    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1824e39-53e4-49bd-b3ae-a7843710a539-tls-certs\") pod \"a1824e39-53e4-49bd-b3ae-a7843710a539\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") "
Sep 15 19:18:12.859857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.859783    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl7xl\" (UniqueName: \"kubernetes.io/projected/a1824e39-53e4-49bd-b3ae-a7843710a539-kube-api-access-bl7xl\") pod \"a1824e39-53e4-49bd-b3ae-a7843710a539\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") "
Sep 15 19:18:12.859857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.859814    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-model-cache\") pod \"a1824e39-53e4-49bd-b3ae-a7843710a539\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") "
Sep 15 19:18:12.860027 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.859859    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-tmp-dir\") pod \"a1824e39-53e4-49bd-b3ae-a7843710a539\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") "
Sep 15 19:18:12.860027 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.859999    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-home\") pod \"a1824e39-53e4-49bd-b3ae-a7843710a539\" (UID: \"a1824e39-53e4-49bd-b3ae-a7843710a539\") "
Sep 15 19:18:12.860151 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.860122    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1824e39-53e4-49bd-b3ae-a7843710a539" (UID: "a1824e39-53e4-49bd-b3ae-a7843710a539"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:12.860219 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.860154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1824e39-53e4-49bd-b3ae-a7843710a539" (UID: "a1824e39-53e4-49bd-b3ae-a7843710a539"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:12.860279 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.860232    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-home" (OuterVolumeSpecName: "home") pod "a1824e39-53e4-49bd-b3ae-a7843710a539" (UID: "a1824e39-53e4-49bd-b3ae-a7843710a539"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:12.860501 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.860485    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:12.860558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.860504    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:12.860558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.860514    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:12.861967 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.861946    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-dshm" (OuterVolumeSpecName: "dshm") pod "a1824e39-53e4-49bd-b3ae-a7843710a539" (UID: "a1824e39-53e4-49bd-b3ae-a7843710a539"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:12.862064 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.862002    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1824e39-53e4-49bd-b3ae-a7843710a539-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1824e39-53e4-49bd-b3ae-a7843710a539" (UID: "a1824e39-53e4-49bd-b3ae-a7843710a539"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:18:12.862064 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.862042    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1824e39-53e4-49bd-b3ae-a7843710a539-kube-api-access-bl7xl" (OuterVolumeSpecName: "kube-api-access-bl7xl") pod "a1824e39-53e4-49bd-b3ae-a7843710a539" (UID: "a1824e39-53e4-49bd-b3ae-a7843710a539"). InnerVolumeSpecName "kube-api-access-bl7xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:12.961984 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.961947    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl7xl\" (UniqueName: \"kubernetes.io/projected/a1824e39-53e4-49bd-b3ae-a7843710a539-kube-api-access-bl7xl\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:12.961984 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.961978    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1824e39-53e4-49bd-b3ae-a7843710a539-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:12.961984 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:12.961988    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1824e39-53e4-49bd-b3ae-a7843710a539-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:13.151029 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:13.150969    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"]
Sep 15 19:18:13.153303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:13.153272    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-17c562de/e2e-lifecycle-model-17c562de-kserve-64c74b7ff9-86j6z"]
Sep 15 19:18:14.234262 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:14.234226    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1824e39-53e4-49bd-b3ae-a7843710a539" path="/var/lib/kubelet/pods/a1824e39-53e4-49bd-b3ae-a7843710a539/volumes"
Sep 15 19:18:14.942545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:14.942510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:20.063257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:20.063216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:20.165434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:20.165399    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:20.694571 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:20.694525    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:24.207891 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:24.207859    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:24.308593 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:24.308561    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:33.837911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:33.837872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:34.796579 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:34.796533    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:34.902424 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:34.902391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:40.681977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:40.681937    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:43.734888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:43.734847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:46.480471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:46.480439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:46.581058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:46.581005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:46.712026 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:46.711994    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"]
Sep 15 19:18:46.712240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:46.712215    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" containerID="cri-o://f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a" gracePeriod=30
Sep 15 19:18:46.730019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:46.729989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:48.011921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:48.011862    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:48.112022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:48.111996    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:48.216325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:48.216283    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:48.597285 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:48.597249    2584 scope.go:117] "RemoveContainer" containerID="c42e24fea382b2353154e06c9ae3025f61a831811305e193653805cd7af30675"
Sep 15 19:18:48.815373 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:48.815345    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:48.917834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:48.917722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:49.020057 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:49.020025    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:49.068545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:49.068513    2584 patch_prober.go:28] interesting pod/e2e-mt-17c562de-openshift-default-88f855448-8xcsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:18:49.068734 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:49.068582    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:18:50.196471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:50.196440    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:51.977770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:51.977714    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:18:52.082254 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-envoy\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082254 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082255    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-credential-socket\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082279    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-socket\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082332    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-certs\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082361    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istiod-ca-cert\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082377    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-data\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082420    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-token\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082465    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97mbw\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-kube-api-access-97mbw\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-podinfo\") pod \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\" (UID: \"5ecb75a1-deb7-4b19-aa84-d3e71542c97e\") "
Sep 15 19:18:52.082909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082550    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:52.082909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082664    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:52.082909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082688    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:52.082909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082705    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:52.082909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.082766    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:18:52.083161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.083071    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.083161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.083088    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.083161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.083097    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.083161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.083105    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.083161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.083117    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.084690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.084653    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:52.084820 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.084776    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-kube-api-access-97mbw" (OuterVolumeSpecName: "kube-api-access-97mbw") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "kube-api-access-97mbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:18:52.084921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.084901    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:18:52.084973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.084959    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ecb75a1-deb7-4b19-aa84-d3e71542c97e" (UID: "5ecb75a1-deb7-4b19-aa84-d3e71542c97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:18:52.099789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.099759    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerID="f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a" exitCode=0
Sep 15 19:18:52.099910 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.099809    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"
Sep 15 19:18:52.099910 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.099835    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" event={"ID":"5ecb75a1-deb7-4b19-aa84-d3e71542c97e","Type":"ContainerDied","Data":"f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a"}
Sep 15 19:18:52.100019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.099925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb" event={"ID":"5ecb75a1-deb7-4b19-aa84-d3e71542c97e","Type":"ContainerDied","Data":"a9742a6d75437df2c66192044dc6f1a5198ce8fd5136e7e4945794a895e3f3f0"}
Sep 15 19:18:52.100019 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.099949    2584 scope.go:117] "RemoveContainer" containerID="f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a"
Sep 15 19:18:52.110473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.110455    2584 scope.go:117] "RemoveContainer" containerID="da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"
Sep 15 19:18:52.119339 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.119295    2584 scope.go:117] "RemoveContainer" containerID="f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a"
Sep 15 19:18:52.119802 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:18:52.119772    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a\": container with ID starting with f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a not found: ID does not exist" containerID="f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a"
Sep 15 19:18:52.119909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.119811    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a"} err="failed to get container status \"f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a\": rpc error: code = NotFound desc = could not find container \"f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a\": container with ID starting with f55b7fbe0766465b385cea3270afe41fe22f08f56569c0214efb80fbb74ac48a not found: ID does not exist"
Sep 15 19:18:52.119909 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.119835    2584 scope.go:117] "RemoveContainer" containerID="da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"
Sep 15 19:18:52.120192 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:18:52.120153    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be\": container with ID starting with da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be not found: ID does not exist" containerID="da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"
Sep 15 19:18:52.120257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.120184    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be"} err="failed to get container status \"da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be\": rpc error: code = NotFound desc = could not find container \"da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be\": container with ID starting with da1b860c5621a487c4f859e139d00151dcf3f264d23d57062b96f7a6577f52be not found: ID does not exist"
Sep 15 19:18:52.123533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.123508    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"]
Sep 15 19:18:52.126542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.126519    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-17c562de-openshift-default-88f855448-8xcsb"]
Sep 15 19:18:52.183966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.183930    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.183966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.183959    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.183966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.183969    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97mbw\" (UniqueName: \"kubernetes.io/projected/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-kube-api-access-97mbw\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.184196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.183979    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecb75a1-deb7-4b19-aa84-d3e71542c97e-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:18:52.235429 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:52.235343    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" path="/var/lib/kubelet/pods/5ecb75a1-deb7-4b19-aa84-d3e71542c97e/volumes"
Sep 15 19:18:55.915342 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:55.915306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:56.110982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:56.110948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:18:56.559160 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:18:56.559125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:00.617845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:00.617811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:02.214256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:02.214224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:02.272741 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:02.272710    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:02.352258 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:02.352218    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:02.454370 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:02.454332    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:03.079506 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:03.079476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:12.882464 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:12.882429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:13.099603 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:13.099570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:14.785799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:14.785752    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:14.886591 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:14.886557    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:15.905350 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:15.905319    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:33.999070 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:33.999040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:39.798511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:39.798478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:39.858088 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:39.858050    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:40.142052 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.141976    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:40.405873 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.405787    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"]
Sep 15 19:19:40.406226 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406213    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3f7720-a22e-474e-9a46-d8f229165c3e" containerName="revoke-keys"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406227    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3f7720-a22e-474e-9a46-d8f229165c3e" containerName="revoke-keys"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406243    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1824e39-53e4-49bd-b3ae-a7843710a539" containerName="main"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406248    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1824e39-53e4-49bd-b3ae-a7843710a539" containerName="main"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406256    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5889c3-c6bf-4403-a52d-35c497b0ebe9" containerName="payload-processing"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406262    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5889c3-c6bf-4403-a52d-35c497b0ebe9" containerName="payload-processing"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406269    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406274    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406284    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d83151e-d37c-42e9-b9f4-17411dbe0ca3" containerName="main"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406288    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d83151e-d37c-42e9-b9f4-17411dbe0ca3" containerName="main"
Sep 15 19:19:40.406296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406297    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406306    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406315    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406320    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406329    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406334    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406396    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d83151e-d37c-42e9-b9f4-17411dbe0ca3" containerName="main"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406402    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406411    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1824e39-53e4-49bd-b3ae-a7843710a539" containerName="main"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406417    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3f7720-a22e-474e-9a46-d8f229165c3e" containerName="revoke-keys"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406426    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406431    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5889c3-c6bf-4403-a52d-35c497b0ebe9" containerName="payload-processing"
Sep 15 19:19:40.406826 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.406437    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e57c12e-68ef-4451-9be0-7a4a9576c8d0" containerName="istio-proxy"
Sep 15 19:19:40.409672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.409651    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.412685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.412661    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-759580-openshift-default-dockercfg-x6g6b\""
Sep 15 19:19:40.440356 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.440331    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"]
Sep 15 19:19:40.447586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.447564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.509935    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-socket\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510007    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-podinfo\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-certs\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-token\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceefe71-9eda-484a-9d5a-6b629816e161-istiod-ca-cert\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-credential-socket\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwxn\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-kube-api-access-lcwxn\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-data\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.513642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.510324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-envoy\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.611554 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwxn\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-kube-api-access-lcwxn\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.611774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-data\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.611774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611597    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-envoy\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.611774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-socket\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.611774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-podinfo\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.611774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-certs\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.611965    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-token\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612119 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-data\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612119 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612076    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-socket\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612126    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-certs\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612323 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceefe71-9eda-484a-9d5a-6b629816e161-istiod-ca-cert\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-credential-socket\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-credential-socket\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.612945 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.612926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceefe71-9eda-484a-9d5a-6b629816e161-istiod-ca-cert\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.614137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.614112    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-envoy\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.614483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.614461    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-podinfo\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.619372 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.619347    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-token\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.619514 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.619497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwxn\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-kube-api-access-lcwxn\") pod \"e2e-api-a-759580-openshift-default-56c58646bd-42hpn\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") " pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:40.707459 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.707381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:40.720664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:40.720605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:41.005946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.005923    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"]
Sep 15 19:19:41.006647 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:19:41.006595    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceefe71_9eda_484a_9d5a_6b629816e161.slice/crio-208ad610650ea021455e31f0acee2ea586f8ed6d3d617b8f56d75d78eca044cf WatchSource:0}: Error finding container 208ad610650ea021455e31f0acee2ea586f8ed6d3d617b8f56d75d78eca044cf: Status 404 returned error can't find the container with id 208ad610650ea021455e31f0acee2ea586f8ed6d3d617b8f56d75d78eca044cf
Sep 15 19:19:41.009059 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.009021    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:19:41.009169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.009103    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:19:41.009169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.009154    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:19:41.431813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.431715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" event={"ID":"5ceefe71-9eda-484a-9d5a-6b629816e161","Type":"ContainerStarted","Data":"05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"}
Sep 15 19:19:41.431813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.431771    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" event={"ID":"5ceefe71-9eda-484a-9d5a-6b629816e161","Type":"ContainerStarted","Data":"208ad610650ea021455e31f0acee2ea586f8ed6d3d617b8f56d75d78eca044cf"}
Sep 15 19:19:41.452302 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.452255    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podStartSLOduration=1.45224178 podStartE2EDuration="1.45224178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:41.449409044 +0000 UTC m=+1733.603777511" watchObservedRunningTime="2026-09-15 19:19:41.45224178 +0000 UTC m=+1733.606610301"
Sep 15 19:19:41.721365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:41.721285    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:42.632866 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:42.632833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:42.727171 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:42.727136    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:43.328819 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:43.328777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:43.427945 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:43.427915    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:43.449213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:43.449178    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:43.455257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:43.455222    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:19:44.127979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:44.127948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:44.809674 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:44.809643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:45.530805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:45.530772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:46.166249 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:46.166220    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:46.835728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:46.835694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:46.891230 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:46.891198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:46.991748 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:46.991717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:47.315279 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:47.315250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:47.386575 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:47.386549    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:56.393325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.393295    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:56.507151 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.507119    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"]
Sep 15 19:19:56.507647 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.507606    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecb75a1-deb7-4b19-aa84-d3e71542c97e" containerName="istio-proxy"
Sep 15 19:19:56.510686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.510670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:19:56.513011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.512992    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:19:56.513115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.513055    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:19:56.526431 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.520502    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"]
Sep 15 19:19:56.680648 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.680555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz\" (UID: \"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:19:56.782150 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.782120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz\" (UID: \"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:19:56.782729 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.782710    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz\" (UID: \"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:19:56.820758 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.820733    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:56.828659 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:56.828638    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:19:57.110858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:57.110830    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"]
Sep 15 19:19:57.111461 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:19:57.111438    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ecb951_9c17_4dad_b5db_4bfa45a1d5be.slice/crio-2d0241c2973b56caacc3071957a4984dc945a0226fa12cbd0dc3f08a2f8d45e8 WatchSource:0}: Error finding container 2d0241c2973b56caacc3071957a4984dc945a0226fa12cbd0dc3f08a2f8d45e8: Status 404 returned error can't find the container with id 2d0241c2973b56caacc3071957a4984dc945a0226fa12cbd0dc3f08a2f8d45e8
Sep 15 19:19:57.503789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:57.503762    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:57.544434 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:57.544405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz" event={"ID":"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be","Type":"ContainerStarted","Data":"9bdf538ad89a028a5c3ed4321d10cc4b283b6e8f2308d08c1403c7d8557da805"}
Sep 15 19:19:57.544546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:57.544465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz" event={"ID":"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be","Type":"ContainerStarted","Data":"2d0241c2973b56caacc3071957a4984dc945a0226fa12cbd0dc3f08a2f8d45e8"}
Sep 15 19:19:57.562759 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:57.562716    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz" podStartSLOduration=1.5626999050000001 podStartE2EDuration="1.562699905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:19:57.560211515 +0000 UTC m=+1749.714579982" watchObservedRunningTime="2026-09-15 19:19:57.562699905 +0000 UTC m=+1749.717068399"
Sep 15 19:19:58.219677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:58.219567    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:58.552491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:58.552455    2584 generic.go:356] "Generic (PLEG): container finished" podID="b3ecb951-9c17-4dad-b5db-4bfa45a1d5be" containerID="9bdf538ad89a028a5c3ed4321d10cc4b283b6e8f2308d08c1403c7d8557da805" exitCode=0
Sep 15 19:19:58.552879 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:58.552542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz" event={"ID":"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be","Type":"ContainerDied","Data":"9bdf538ad89a028a5c3ed4321d10cc4b283b6e8f2308d08c1403c7d8557da805"}
Sep 15 19:19:58.959527 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:58.959445    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:59.626776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:59.626743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:19:59.703057 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:59.703031    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:19:59.822897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:59.822800    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be-maas-api-service-ca\") pod \"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be\" (UID: \"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be\") "
Sep 15 19:19:59.823163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:59.823140    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3ecb951-9c17-4dad-b5db-4bfa45a1d5be" (UID: "b3ecb951-9c17-4dad-b5db-4bfa45a1d5be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:19:59.924897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:19:59.924859    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:00.428784 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:00.428727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:00.566677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:00.566647    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"
Sep 15 19:20:00.566677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:00.566644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz" event={"ID":"b3ecb951-9c17-4dad-b5db-4bfa45a1d5be","Type":"ContainerDied","Data":"2d0241c2973b56caacc3071957a4984dc945a0226fa12cbd0dc3f08a2f8d45e8"}
Sep 15 19:20:00.566952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:00.566712    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0241c2973b56caacc3071957a4984dc945a0226fa12cbd0dc3f08a2f8d45e8"
Sep 15 19:20:01.205537 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:01.205500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:01.397376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:01.397342    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"]
Sep 15 19:20:01.407265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:01.407227    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"]
Sep 15 19:20:01.466601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:01.466497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:01.993664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:01.993632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:02.866795 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:02.866702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:03.120740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.120660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:03.420490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.420390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:03.546239 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.546202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:03.794498 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.794462    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"]
Sep 15 19:20:03.794987 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.794973    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ecb951-9c17-4dad-b5db-4bfa45a1d5be" containerName="revoke-keys"
Sep 15 19:20:03.795058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.794990    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ecb951-9c17-4dad-b5db-4bfa45a1d5be" containerName="revoke-keys"
Sep 15 19:20:03.795113 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.795076    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ecb951-9c17-4dad-b5db-4bfa45a1d5be" containerName="revoke-keys"
Sep 15 19:20:03.798384 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.798362    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.800809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.800788    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-16a967-openshift-default-dockercfg-scwl6\""
Sep 15 19:20:03.812418 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.812396    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"]
Sep 15 19:20:03.825972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.825948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:03.865705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865681    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-certs\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.865824 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865709    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istiod-ca-cert\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.865824 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-podinfo\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.865824 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplft\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-kube-api-access-lplft\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.865824 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-envoy\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.865970 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865842    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-credential-socket\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.865970 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-token\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.866038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.865966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-socket\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.866038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.866023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-data\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-certs\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967327    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istiod-ca-cert\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-podinfo\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplft\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-kube-api-access-lplft\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967567    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-envoy\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-credential-socket\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-token\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-socket\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-certs\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.967886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.967844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-data\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.968291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.968133    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-socket\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.968291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.968174    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istiod-ca-cert\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.968291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.968178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-credential-socket\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.968291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.968274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-data\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.969858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.969839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-envoy\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.970138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.970122    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-podinfo\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.976660 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.976598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-token\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:03.977113 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:03.977091    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplft\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-kube-api-access-lplft\") pod \"e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") " pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:04.006622 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.006581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:04.111951 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.111874    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:04.130397 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.130372    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:04.409741 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.409713    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"]
Sep 15 19:20:04.415883 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.415849    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:04.415968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.415914    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:04.415968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.415943    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:04.596306 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.596269    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" event={"ID":"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6","Type":"ContainerStarted","Data":"e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"}
Sep 15 19:20:04.596502 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.596328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" event={"ID":"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6","Type":"ContainerStarted","Data":"0d823c772522a07c81e6d603c8b1d9fc02389c1931edd73ab25969aab1c54b4b"}
Sep 15 19:20:04.648437 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:04.648380    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podStartSLOduration=1.648357169 podStartE2EDuration="1.648357169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:20:04.643604136 +0000 UTC m=+1756.797972607" watchObservedRunningTime="2026-09-15 19:20:04.648357169 +0000 UTC m=+1756.802725632"
Sep 15 19:20:05.073732 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:05.073700    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:05.112591 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:05.112565    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:05.117510 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:05.117486    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:05.603591 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:05.603564    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:05.604588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:05.604564    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:05.826015 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:05.825983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:06.546376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:06.546342    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:06.642540 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:06.642505    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:07.624036 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:07.624002    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:08.518287 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:08.518257    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:09.146465 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:09.146432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:09.927968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:09.927938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:10.474292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:10.474260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:11.154028 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:11.153997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:11.848658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:11.848601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:11.903781 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:11.903753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.028948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.028902    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.280303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.280276    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.380314 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.380282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.385494 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.385467    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"]
Sep 15 19:20:12.390463 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.390447    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.392651 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.392633    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-16a967\""
Sep 15 19:20:12.392755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.392731    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-16a967-dockercfg-989tg\""
Sep 15 19:20:12.399039 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.399018    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"]
Sep 15 19:20:12.475157 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.475125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57339a50-b218-4ba6-82dc-8d8d44702b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.475365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.475201    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqnk\" (UniqueName: \"kubernetes.io/projected/57339a50-b218-4ba6-82dc-8d8d44702b2c-kube-api-access-rhqnk\") pod \"payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.577056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.576791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.578362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.578328    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57339a50-b218-4ba6-82dc-8d8d44702b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.578604 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.578587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqnk\" (UniqueName: \"kubernetes.io/projected/57339a50-b218-4ba6-82dc-8d8d44702b2c-kube-api-access-rhqnk\") pod \"payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.580092 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.580066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57339a50-b218-4ba6-82dc-8d8d44702b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.590333 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.590294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqnk\" (UniqueName: \"kubernetes.io/projected/57339a50-b218-4ba6-82dc-8d8d44702b2c-kube-api-access-rhqnk\") pod \"payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.644159 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.644077    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.703461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.703421    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:12.703727 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.703702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.772104 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.772045    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"]
Sep 15 19:20:12.775342 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.775307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:12.779733 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:12.779709    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-900c88-ggtxz"]
Sep 15 19:20:13.010485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.010459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"]
Sep 15 19:20:13.012029 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:20:13.011993    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57339a50_b218_4ba6_82dc_8d8d44702b2c.slice/crio-8177bbe034da4e4e83e32e3956eb2437648f6ece6fffd382c14b0bf8110c4d5b WatchSource:0}: Error finding container 8177bbe034da4e4e83e32e3956eb2437648f6ece6fffd382c14b0bf8110c4d5b: Status 404 returned error can't find the container with id 8177bbe034da4e4e83e32e3956eb2437648f6ece6fffd382c14b0bf8110c4d5b
Sep 15 19:20:13.135325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.135288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:13.412871 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.412788    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:13.678850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.678765    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl" event={"ID":"57339a50-b218-4ba6-82dc-8d8d44702b2c","Type":"ContainerStarted","Data":"5d4f4f575eaf1858fa24ce4da81ca40c57ed5acab65674b5d41d6b114df6b3bb"}
Sep 15 19:20:13.678850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.678814    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl" event={"ID":"57339a50-b218-4ba6-82dc-8d8d44702b2c","Type":"ContainerStarted","Data":"8177bbe034da4e4e83e32e3956eb2437648f6ece6fffd382c14b0bf8110c4d5b"}
Sep 15 19:20:13.678850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.678847    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:13.706827 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:13.706774    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl" podStartSLOduration=1.706758722 podStartE2EDuration="1.706758722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:20:13.705081046 +0000 UTC m=+1765.859449514" watchObservedRunningTime="2026-09-15 19:20:13.706758722 +0000 UTC m=+1765.861127189"
Sep 15 19:20:14.152123 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:14.152093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:14.238219 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:14.238188    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ecb951-9c17-4dad-b5db-4bfa45a1d5be" path="/var/lib/kubelet/pods/b3ecb951-9c17-4dad-b5db-4bfa45a1d5be/volumes"
Sep 15 19:20:14.823361 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:14.823293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:15.624538 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:15.624501    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:16.367112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.367076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:16.418512 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.418463    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s" podUID="efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" containerName="main" containerID="cri-o://e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f" gracePeriod=45
Sep 15 19:20:16.434201 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.434155    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d" podUID="999e7660-0f32-489b-9d36-1d84211bfd55" containerName="main" containerID="cri-o://e4df411d88b3f2134126dbb0acff6ce1150964c091b778003759769d6ef0decc" gracePeriod=45
Sep 15 19:20:16.715742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.715334    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:20:16.715742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.715502    2584 generic.go:356] "Generic (PLEG): container finished" podID="efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" containerID="e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f" exitCode=0
Sep 15 19:20:16.715742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.715731    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s" event={"ID":"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6","Type":"ContainerDied","Data":"e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f"}
Sep 15 19:20:16.716540 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.715770    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s" event={"ID":"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6","Type":"ContainerDied","Data":"056b0a4b65fafb460b32d87080b4770c25127014662cbb94703c37ec826b6728"}
Sep 15 19:20:16.716540 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.715792    2584 scope.go:117] "RemoveContainer" containerID="e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f"
Sep 15 19:20:16.719718 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.719549    2584 generic.go:356] "Generic (PLEG): container finished" podID="999e7660-0f32-489b-9d36-1d84211bfd55" containerID="e4df411d88b3f2134126dbb0acff6ce1150964c091b778003759769d6ef0decc" exitCode=0
Sep 15 19:20:16.719718 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.719634    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d" event={"ID":"999e7660-0f32-489b-9d36-1d84211bfd55","Type":"ContainerDied","Data":"e4df411d88b3f2134126dbb0acff6ce1150964c091b778003759769d6ef0decc"}
Sep 15 19:20:16.734831 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.734690    2584 scope.go:117] "RemoveContainer" containerID="e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f"
Sep 15 19:20:16.735570 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:20:16.735470    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f\": container with ID starting with e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f not found: ID does not exist" containerID="e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f"
Sep 15 19:20:16.735570 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.735513    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f"} err="failed to get container status \"e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f\": rpc error: code = NotFound desc = could not find container \"e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f\": container with ID starting with e30d1982ef348032589cc7e738fb8110e28cbaa1146958a25d9ae4ebd682d51f not found: ID does not exist"
Sep 15 19:20:16.760635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.760586    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:20:16.841816 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.841774    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tmp-dir\") pod \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") "
Sep 15 19:20:16.842010 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.841884    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-dshm\") pod \"999e7660-0f32-489b-9d36-1d84211bfd55\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") "
Sep 15 19:20:16.842010 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.841926    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/999e7660-0f32-489b-9d36-1d84211bfd55-tls-certs\") pod \"999e7660-0f32-489b-9d36-1d84211bfd55\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") "
Sep 15 19:20:16.842122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842033    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-dshm\") pod \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") "
Sep 15 19:20:16.842122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842083    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-tmp-dir\") pod \"999e7660-0f32-489b-9d36-1d84211bfd55\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") "
Sep 15 19:20:16.842122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842093    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" (UID: "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.842122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842112    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmdqs\" (UniqueName: \"kubernetes.io/projected/999e7660-0f32-489b-9d36-1d84211bfd55-kube-api-access-fmdqs\") pod \"999e7660-0f32-489b-9d36-1d84211bfd55\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") "
Sep 15 19:20:16.842341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842183    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zf62\" (UniqueName: \"kubernetes.io/projected/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-kube-api-access-6zf62\") pod \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") "
Sep 15 19:20:16.842341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842219    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-home\") pod \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") "
Sep 15 19:20:16.842341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842255    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tls-certs\") pod \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") "
Sep 15 19:20:16.842341 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-model-cache\") pod \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\" (UID: \"efa122f1-f8ef-43db-8a25-b0ee0e8da2c6\") "
Sep 15 19:20:16.842538 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842350    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "999e7660-0f32-489b-9d36-1d84211bfd55" (UID: "999e7660-0f32-489b-9d36-1d84211bfd55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.842538 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842362    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-home\") pod \"999e7660-0f32-489b-9d36-1d84211bfd55\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") "
Sep 15 19:20:16.842538 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842425    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-model-cache\") pod \"999e7660-0f32-489b-9d36-1d84211bfd55\" (UID: \"999e7660-0f32-489b-9d36-1d84211bfd55\") "
Sep 15 19:20:16.842701 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842538    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-home" (OuterVolumeSpecName: "home") pod "999e7660-0f32-489b-9d36-1d84211bfd55" (UID: "999e7660-0f32-489b-9d36-1d84211bfd55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.842844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842747    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" (UID: "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.843042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842876    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-home" (OuterVolumeSpecName: "home") pod "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" (UID: "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.843042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.842907    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-model-cache" (OuterVolumeSpecName: "model-cache") pod "999e7660-0f32-489b-9d36-1d84211bfd55" (UID: "999e7660-0f32-489b-9d36-1d84211bfd55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.843518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.843418    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.843518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.843445    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.843518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.843459    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.843518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.843474    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.843518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.843486    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.843518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.843498    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.845450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.845414    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999e7660-0f32-489b-9d36-1d84211bfd55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "999e7660-0f32-489b-9d36-1d84211bfd55" (UID: "999e7660-0f32-489b-9d36-1d84211bfd55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:16.845561 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.845458    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" (UID: "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:20:16.845679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.845658    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-dshm" (OuterVolumeSpecName: "dshm") pod "999e7660-0f32-489b-9d36-1d84211bfd55" (UID: "999e7660-0f32-489b-9d36-1d84211bfd55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.845805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.845783    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-dshm" (OuterVolumeSpecName: "dshm") pod "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" (UID: "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:16.846036 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.846001    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-kube-api-access-6zf62" (OuterVolumeSpecName: "kube-api-access-6zf62") pod "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" (UID: "efa122f1-f8ef-43db-8a25-b0ee0e8da2c6"). InnerVolumeSpecName "kube-api-access-6zf62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:16.846505 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.846476    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999e7660-0f32-489b-9d36-1d84211bfd55-kube-api-access-fmdqs" (OuterVolumeSpecName: "kube-api-access-fmdqs") pod "999e7660-0f32-489b-9d36-1d84211bfd55" (UID: "999e7660-0f32-489b-9d36-1d84211bfd55"). InnerVolumeSpecName "kube-api-access-fmdqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:16.944666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.944560    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/999e7660-0f32-489b-9d36-1d84211bfd55-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.944666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.944591    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/999e7660-0f32-489b-9d36-1d84211bfd55-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.944666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.944603    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.944666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.944638    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdqs\" (UniqueName: \"kubernetes.io/projected/999e7660-0f32-489b-9d36-1d84211bfd55-kube-api-access-fmdqs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.944666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.944647    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf62\" (UniqueName: \"kubernetes.io/projected/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-kube-api-access-6zf62\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:16.944666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:16.944658    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:17.057535 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.057502    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:17.727920 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.727838    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"
Sep 15 19:20:17.727920 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.727875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d" event={"ID":"999e7660-0f32-489b-9d36-1d84211bfd55","Type":"ContainerDied","Data":"ac9ca0c712a077478e971713a69969252bb8bac3eba0dd7be7aebda445ca724b"}
Sep 15 19:20:17.728467 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.727931    2584 scope.go:117] "RemoveContainer" containerID="e4df411d88b3f2134126dbb0acff6ce1150964c091b778003759769d6ef0decc"
Sep 15 19:20:17.729527 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.729069    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"
Sep 15 19:20:17.731453 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.731420    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerID="05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1" exitCode=0
Sep 15 19:20:17.731593 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.731508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" event={"ID":"5ceefe71-9eda-484a-9d5a-6b629816e161","Type":"ContainerDied","Data":"05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"}
Sep 15 19:20:17.732052 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.732036    2584 scope.go:117] "RemoveContainer" containerID="05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"
Sep 15 19:20:17.733512 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.733448    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:17.733663 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.733537    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:17.733663 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.733600    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:17.768332 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.768297    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"]
Sep 15 19:20:17.773045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.773018    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/premium-w0-900c88-kserve-78bdb5c5b8-bxw6s"]
Sep 15 19:20:17.782666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.782630    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"]
Sep 15 19:20:17.787418 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.787395    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-900c88/simulator-w0-900c88-kserve-57ffb4869c-95w4d"]
Sep 15 19:20:17.924687 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:17.924630    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:18.234810 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:18.234778    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="999e7660-0f32-489b-9d36-1d84211bfd55" path="/var/lib/kubelet/pods/999e7660-0f32-489b-9d36-1d84211bfd55/volumes"
Sep 15 19:20:18.235210 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:18.235197    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" path="/var/lib/kubelet/pods/efa122f1-f8ef-43db-8a25-b0ee0e8da2c6/volumes"
Sep 15 19:20:18.740798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:18.740761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" event={"ID":"5ceefe71-9eda-484a-9d5a-6b629816e161","Type":"ContainerStarted","Data":"1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188"}
Sep 15 19:20:18.975217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:18.975189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:19.050628 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.050566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:19.239056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.239023    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"]
Sep 15 19:20:19.239276 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.239250    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy" containerID="cri-o://8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d" gracePeriod=30
Sep 15 19:20:19.253430 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.253400    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:19.721356 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.721314    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:20:19.721556 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.721511    2584 patch_prober.go:28] interesting pod/e2e-api-a-759580-openshift-default-56c58646bd-42hpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 19:20:19.721638 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:19.721571    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 19:20:20.721894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:20.721857    2584 patch_prober.go:28] interesting pod/e2e-api-a-759580-openshift-default-56c58646bd-42hpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 19:20:20.722473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:20.722441    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 19:20:21.721527 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:21.721491    2584 patch_prober.go:28] interesting pod/e2e-api-a-759580-openshift-default-56c58646bd-42hpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 19:20:21.721861 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:21.721816    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 19:20:22.721326 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:22.721285    2584 patch_prober.go:28] interesting pod/e2e-api-a-759580-openshift-default-56c58646bd-42hpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 19:20:22.721835 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:22.721366    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 19:20:23.721638 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:23.721577    2584 patch_prober.go:28] interesting pod/e2e-api-a-759580-openshift-default-56c58646bd-42hpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 19:20:23.722099 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:23.721663    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 19:20:24.515340 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.515287    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45676d20-747e-46f4-8ef5-d3a9dd784777-istiod-ca-cert\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646094    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-socket\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646207    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-certs\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646238    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-data\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646297    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-envoy\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646351    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-credential-socket\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646388    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-token\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646434    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddvv8\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-kube-api-access-ddvv8\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646463    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-podinfo\") pod \"45676d20-747e-46f4-8ef5-d3a9dd784777\" (UID: \"45676d20-747e-46f4-8ef5-d3a9dd784777\") "
Sep 15 19:20:24.646978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.646893    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-data" (OuterVolumeSpecName: "istio-data") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:24.647705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.647244    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45676d20-747e-46f4-8ef5-d3a9dd784777-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:24.647705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.647388    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:24.647705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.647511    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:24.647705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.647674    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:24.650674 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.650355    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-token" (OuterVolumeSpecName: "istio-token") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:24.650914 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.650887    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:20:24.651510 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.651486    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:20:24.652301 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.652280    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-kube-api-access-ddvv8" (OuterVolumeSpecName: "kube-api-access-ddvv8") pod "45676d20-747e-46f4-8ef5-d3a9dd784777" (UID: "45676d20-747e-46f4-8ef5-d3a9dd784777"). InnerVolumeSpecName "kube-api-access-ddvv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:20:24.688503 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.688484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:20:24.724639 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.721361    2584 patch_prober.go:28] interesting pod/e2e-api-a-759580-openshift-default-56c58646bd-42hpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 19:20:24.724639 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.721427    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751350    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751385    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751398    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751411    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751424    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751437    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddvv8\" (UniqueName: \"kubernetes.io/projected/45676d20-747e-46f4-8ef5-d3a9dd784777-kube-api-access-ddvv8\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751450    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45676d20-747e-46f4-8ef5-d3a9dd784777-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751462    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45676d20-747e-46f4-8ef5-d3a9dd784777-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.756635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.751476    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45676d20-747e-46f4-8ef5-d3a9dd784777-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:24.788829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.788732    2584 generic.go:356] "Generic (PLEG): container finished" podID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerID="8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d" exitCode=0
Sep 15 19:20:24.788829 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.788810    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"
Sep 15 19:20:24.789012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.788823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" event={"ID":"45676d20-747e-46f4-8ef5-d3a9dd784777","Type":"ContainerDied","Data":"8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d"}
Sep 15 19:20:24.789012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.788881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld" event={"ID":"45676d20-747e-46f4-8ef5-d3a9dd784777","Type":"ContainerDied","Data":"d59ea3ccf9722dcdc9857798abb7c5c7880ade226d151a17076c52b736d9865a"}
Sep 15 19:20:24.789012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.788907    2584 scope.go:117] "RemoveContainer" containerID="8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d"
Sep 15 19:20:24.799417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.799398    2584 scope.go:117] "RemoveContainer" containerID="b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"
Sep 15 19:20:24.808103 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.808080    2584 scope.go:117] "RemoveContainer" containerID="8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d"
Sep 15 19:20:24.808403 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:20:24.808379    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d\": container with ID starting with 8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d not found: ID does not exist" containerID="8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d"
Sep 15 19:20:24.808471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.808415    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d"} err="failed to get container status \"8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d\": rpc error: code = NotFound desc = could not find container \"8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d\": container with ID starting with 8cf2784408f4695a256af6695f82bd90eae4bb14f7ffbe4adcabe99aef55856d not found: ID does not exist"
Sep 15 19:20:24.808471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.808442    2584 scope.go:117] "RemoveContainer" containerID="b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"
Sep 15 19:20:24.808811 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:20:24.808784    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2\": container with ID starting with b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2 not found: ID does not exist" containerID="b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"
Sep 15 19:20:24.808911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.808820    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2"} err="failed to get container status \"b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2\": rpc error: code = NotFound desc = could not find container \"b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2\": container with ID starting with b9d0185180dfc665327806bb451199060b40f98d156b075943738838aa0583f2 not found: ID does not exist"
Sep 15 19:20:24.812234 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.812215    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"]
Sep 15 19:20:24.815810 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:24.815791    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-900c88-openshift-default-5c77b9998d-8h5ld"]
Sep 15 19:20:25.760201 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:25.760171    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:20:25.760726 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:25.760597    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:20:25.761433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:25.761411    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:20:26.239462 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:26.239361    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" path="/var/lib/kubelet/pods/45676d20-747e-46f4-8ef5-d3a9dd784777/volumes"
Sep 15 19:20:28.652772 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:28.652735    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:28.718259 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:28.718229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:28.914106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:28.914027    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:33.643716 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.643688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:33.858451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.858360    2584 generic.go:356] "Generic (PLEG): container finished" podID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerID="e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63" exitCode=0
Sep 15 19:20:33.858451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.858437    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" event={"ID":"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6","Type":"ContainerDied","Data":"e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"}
Sep 15 19:20:33.858912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.858884    2584 scope.go:117] "RemoveContainer" containerID="e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"
Sep 15 19:20:33.859995 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.859962    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:33.860097 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.860032    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:33.860097 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:33.860062    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:20:34.214676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:34.214592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:34.868140 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:34.868103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" event={"ID":"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6","Type":"ContainerStarted","Data":"590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8"}
Sep 15 19:20:35.112268 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.112237    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:35.113182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.113156    2584 patch_prober.go:28] interesting pod/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Sep 15 19:20:35.113303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.113201    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Sep 15 19:20:35.125792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.125694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:35.291644 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.291579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:35.603872 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.603839    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:35.870349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:35.870236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:36.112331 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:36.112298    2584 patch_prober.go:28] interesting pod/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Sep 15 19:20:36.112504 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:36.112357    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Sep 15 19:20:36.565485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:36.565455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:36.664512 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:36.664469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:37.112777 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:37.112741    2584 patch_prober.go:28] interesting pod/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Sep 15 19:20:37.113169 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:37.112795    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Sep 15 19:20:37.529080 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:37.529042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:38.112666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:38.112598    2584 patch_prober.go:28] interesting pod/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Sep 15 19:20:38.112861 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:38.112694    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Sep 15 19:20:38.206647 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:38.206551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:39.113300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.113264    2584 patch_prober.go:28] interesting pod/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Sep 15 19:20:39.113715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.113333    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Sep 15 19:20:39.397903 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.396843    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"]
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398498    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" containerName="main"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398524    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" containerName="main"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398546    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="999e7660-0f32-489b-9d36-1d84211bfd55" containerName="main"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398555    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="999e7660-0f32-489b-9d36-1d84211bfd55" containerName="main"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398606    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398632    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398675    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398683    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398888    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398901    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="999e7660-0f32-489b-9d36-1d84211bfd55" containerName="main"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398919    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45676d20-747e-46f4-8ef5-d3a9dd784777" containerName="istio-proxy"
Sep 15 19:20:39.402689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.398933    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa122f1-f8ef-43db-8a25-b0ee0e8da2c6" containerName="main"
Sep 15 19:20:39.404622 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.404585    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:39.407499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.407103    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:20:39.407499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.407116    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:20:39.408127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.408106    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"]
Sep 15 19:20:39.512037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.511999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-759580-pzhzj\" (UID: \"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:39.612815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.612783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-759580-pzhzj\" (UID: \"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:39.613394 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.613373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-759580-pzhzj\" (UID: \"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:39.719568 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.719489    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:39.751980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:39.751948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:40.021805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:40.021773    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"]
Sep 15 19:20:40.022760 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:20:40.022735    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9d6da2_b3c1_402b_9daf_4acdf3c6fb35.slice/crio-37f8d59fc58b5f0011d6add784a461b329c3afe0ef101a30cfde4ca87d3405a6 WatchSource:0}: Error finding container 37f8d59fc58b5f0011d6add784a461b329c3afe0ef101a30cfde4ca87d3405a6: Status 404 returned error can't find the container with id 37f8d59fc58b5f0011d6add784a461b329c3afe0ef101a30cfde4ca87d3405a6
Sep 15 19:20:40.117549 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:40.117520    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:40.118812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:40.118790    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:20:40.910410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:40.910340    2584 generic.go:356] "Generic (PLEG): container finished" podID="1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35" containerID="0f39037757b60f1717ad3881db53785c70cb6cc44f57442c23c1c08f1e258842" exitCode=0
Sep 15 19:20:40.910550 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:40.910423    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj" event={"ID":"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35","Type":"ContainerDied","Data":"0f39037757b60f1717ad3881db53785c70cb6cc44f57442c23c1c08f1e258842"}
Sep 15 19:20:40.910550 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:40.910475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj" event={"ID":"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35","Type":"ContainerStarted","Data":"37f8d59fc58b5f0011d6add784a461b329c3afe0ef101a30cfde4ca87d3405a6"}
Sep 15 19:20:41.082340 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:41.082312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:41.219126 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:41.219089    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:41.758797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:41.758738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:42.056820 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.056797    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:42.141349 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.141315    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35-maas-api-service-ca\") pod \"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35\" (UID: \"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35\") "
Sep 15 19:20:42.141698 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.141676    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35" (UID: "1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:20:42.243146 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.243121    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:20:42.522148 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.522116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:42.925266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.925183    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"
Sep 15 19:20:42.925266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.925216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj" event={"ID":"1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35","Type":"ContainerDied","Data":"37f8d59fc58b5f0011d6add784a461b329c3afe0ef101a30cfde4ca87d3405a6"}
Sep 15 19:20:42.925445 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:42.925278    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f8d59fc58b5f0011d6add784a461b329c3afe0ef101a30cfde4ca87d3405a6"
Sep 15 19:20:43.334380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:43.334350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:44.124879 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:44.124846    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:44.767783 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:44.767748    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:48.654060 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:48.654027    2584 scope.go:117] "RemoveContainer" containerID="17eab260613e0707385a85f9d5106e143094549b42e2167931049401ff983b38"
Sep 15 19:20:55.046565 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.046535    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:55.145379 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.145346    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:55.239485 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.239453    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"]
Sep 15 19:20:55.242956 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.242930    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-759580-pzhzj"]
Sep 15 19:20:55.243526 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.243506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:55.307852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.307781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:55.579653 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.579537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:55.681877 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:55.681833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:56.181406 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:56.181375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:56.234857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:56.234827    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35" path="/var/lib/kubelet/pods/1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35/volumes"
Sep 15 19:20:56.931923 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:56.931885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:57.653461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:57.653420    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:58.423748 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:58.423712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:20:59.232229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:20:59.232194    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:01.609381 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:01.609351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:01.732383 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:01.732350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:01.842079 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:01.842041    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"]
Sep 15 19:21:01.842294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:01.842269    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy" containerID="cri-o://1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188" gracePeriod=30
Sep 15 19:21:01.858592 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:01.858543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:02.175071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.175031    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"]
Sep 15 19:21:02.175483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.175470    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35" containerName="revoke-keys"
Sep 15 19:21:02.175528 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.175484    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35" containerName="revoke-keys"
Sep 15 19:21:02.175573 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.175565    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9d6da2-b3c1-402b-9daf-4acdf3c6fb35" containerName="revoke-keys"
Sep 15 19:21:02.177830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.177815    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:02.180079 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.180056    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:21:02.180209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.180081    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:21:02.186896 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.186834    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"]
Sep 15 19:21:02.252982 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.252952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5394a355-bc69-4c72-b741-7b60fd3d278e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-16a967-kpk8w\" (UID: \"5394a355-bc69-4c72-b741-7b60fd3d278e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:02.334126 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.334085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:02.353667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.353607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5394a355-bc69-4c72-b741-7b60fd3d278e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-16a967-kpk8w\" (UID: \"5394a355-bc69-4c72-b741-7b60fd3d278e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:02.354255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.354229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5394a355-bc69-4c72-b741-7b60fd3d278e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-16a967-kpk8w\" (UID: \"5394a355-bc69-4c72-b741-7b60fd3d278e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:02.490574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.490493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:02.766275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:02.766245    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"]
Sep 15 19:21:02.767672 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:21:02.767645    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5394a355_bc69_4c72_b741_7b60fd3d278e.slice/crio-49da83f95d187659faf2b68dc785c62a29212f229927f1e4075a8483c1d387c6 WatchSource:0}: Error finding container 49da83f95d187659faf2b68dc785c62a29212f229927f1e4075a8483c1d387c6: Status 404 returned error can't find the container with id 49da83f95d187659faf2b68dc785c62a29212f229927f1e4075a8483c1d387c6
Sep 15 19:21:03.062404 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:03.062376    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w" event={"ID":"5394a355-bc69-4c72-b741-7b60fd3d278e","Type":"ContainerStarted","Data":"66a7e2d9db4aaf3751c3576e2e2fedd92bb9b7b54c7913cc5c6c307a8f87e6fe"}
Sep 15 19:21:03.062546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:03.062428    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w" event={"ID":"5394a355-bc69-4c72-b741-7b60fd3d278e","Type":"ContainerStarted","Data":"49da83f95d187659faf2b68dc785c62a29212f229927f1e4075a8483c1d387c6"}
Sep 15 19:21:03.080464 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:03.080416    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w" podStartSLOduration=1.080402235 podStartE2EDuration="1.080402235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:21:03.077507294 +0000 UTC m=+1815.231875763" watchObservedRunningTime="2026-09-15 19:21:03.080402235 +0000 UTC m=+1815.234770702"
Sep 15 19:21:03.156439 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:03.156409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:03.817477 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:03.817390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:04.071247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:04.071153    2584 generic.go:356] "Generic (PLEG): container finished" podID="5394a355-bc69-4c72-b741-7b60fd3d278e" containerID="66a7e2d9db4aaf3751c3576e2e2fedd92bb9b7b54c7913cc5c6c307a8f87e6fe" exitCode=0
Sep 15 19:21:04.071247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:04.071220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w" event={"ID":"5394a355-bc69-4c72-b741-7b60fd3d278e","Type":"ContainerDied","Data":"66a7e2d9db4aaf3751c3576e2e2fedd92bb9b7b54c7913cc5c6c307a8f87e6fe"}
Sep 15 19:21:04.516878 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:04.516844    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:05.186037 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:05.186003    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:05.214850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:05.214829    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:05.283653 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:05.283601    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5394a355-bc69-4c72-b741-7b60fd3d278e-maas-api-service-ca\") pod \"5394a355-bc69-4c72-b741-7b60fd3d278e\" (UID: \"5394a355-bc69-4c72-b741-7b60fd3d278e\") "
Sep 15 19:21:05.283963 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:05.283939    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5394a355-bc69-4c72-b741-7b60fd3d278e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5394a355-bc69-4c72-b741-7b60fd3d278e" (UID: "5394a355-bc69-4c72-b741-7b60fd3d278e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:05.385442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:05.385367    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5394a355-bc69-4c72-b741-7b60fd3d278e-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:05.933587 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:05.933548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:06.091790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:06.091761    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"
Sep 15 19:21:06.091966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:06.091803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w" event={"ID":"5394a355-bc69-4c72-b741-7b60fd3d278e","Type":"ContainerDied","Data":"49da83f95d187659faf2b68dc785c62a29212f229927f1e4075a8483c1d387c6"}
Sep 15 19:21:06.091966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:06.091854    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49da83f95d187659faf2b68dc785c62a29212f229927f1e4075a8483c1d387c6"
Sep 15 19:21:06.555785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:06.555753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:07.094640 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.094597    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:21:07.100183 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.100157    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerID="1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188" exitCode=0
Sep 15 19:21:07.100291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.100196    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"
Sep 15 19:21:07.100291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.100237    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" event={"ID":"5ceefe71-9eda-484a-9d5a-6b629816e161","Type":"ContainerDied","Data":"1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188"}
Sep 15 19:21:07.100369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.100292    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn" event={"ID":"5ceefe71-9eda-484a-9d5a-6b629816e161","Type":"ContainerDied","Data":"208ad610650ea021455e31f0acee2ea586f8ed6d3d617b8f56d75d78eca044cf"}
Sep 15 19:21:07.100369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.100320    2584 scope.go:117] "RemoveContainer" containerID="1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188"
Sep 15 19:21:07.112224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.112188    2584 scope.go:117] "RemoveContainer" containerID="05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"
Sep 15 19:21:07.122484 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.122463    2584 scope.go:117] "RemoveContainer" containerID="1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188"
Sep 15 19:21:07.122831 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:07.122806    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188\": container with ID starting with 1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188 not found: ID does not exist" containerID="1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188"
Sep 15 19:21:07.122953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.122840    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188"} err="failed to get container status \"1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188\": rpc error: code = NotFound desc = could not find container \"1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188\": container with ID starting with 1f1bb061e612ae2b19586382d6f801912374f8015719636dbfc83ca1a680f188 not found: ID does not exist"
Sep 15 19:21:07.122953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.122858    2584 scope.go:117] "RemoveContainer" containerID="05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"
Sep 15 19:21:07.123169 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:07.123152    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1\": container with ID starting with 05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1 not found: ID does not exist" containerID="05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"
Sep 15 19:21:07.123237 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.123173    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1"} err="failed to get container status \"05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1\": rpc error: code = NotFound desc = could not find container \"05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1\": container with ID starting with 05541f12f71d398f84c0ef3fe86fa96d5130df149a4d0f002335e822c5ea26c1 not found: ID does not exist"
Sep 15 19:21:07.205959 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.205939    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-envoy\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206074 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.205986    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-credential-socket\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206074 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-certs\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206192 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206100    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-podinfo\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206192 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206139    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-data\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206192 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206169    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceefe71-9eda-484a-9d5a-6b629816e161-istiod-ca-cert\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206202    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcwxn\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-kube-api-access-lcwxn\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-token\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-socket\") pod \"5ceefe71-9eda-484a-9d5a-6b629816e161\" (UID: \"5ceefe71-9eda-484a-9d5a-6b629816e161\") "
Sep 15 19:21:07.206347 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206264    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:07.206558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206335    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:07.206558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206418    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:07.206558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:07.206752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206564    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ceefe71-9eda-484a-9d5a-6b629816e161-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:07.206815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206796    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.206864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206817    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.206864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206832    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.206864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206846    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceefe71-9eda-484a-9d5a-6b629816e161-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.206864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.206857    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.208272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.208251    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:07.208272 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.208261    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:21:07.208695 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.208664    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:07.208695 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.208673    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-kube-api-access-lcwxn" (OuterVolumeSpecName: "kube-api-access-lcwxn") pod "5ceefe71-9eda-484a-9d5a-6b629816e161" (UID: "5ceefe71-9eda-484a-9d5a-6b629816e161"). InnerVolumeSpecName "kube-api-access-lcwxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:07.264668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.264601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:07.308426 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.308403    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.308426 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.308426    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcwxn\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-kube-api-access-lcwxn\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.308605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.308437    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.308605 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.308446    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceefe71-9eda-484a-9d5a-6b629816e161-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:07.424669 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.424630    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"]
Sep 15 19:21:07.428161 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:07.428139    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-759580-openshift-default-56c58646bd-42hpn"]
Sep 15 19:21:08.235846 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:08.235815    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" path="/var/lib/kubelet/pods/5ceefe71-9eda-484a-9d5a-6b629816e161/volumes"
Sep 15 19:21:12.315011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:12.314979    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:17.926577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:17.926546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:17.962205 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:17.962171    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"]
Sep 15 19:21:17.962453 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:17.962423    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl" podUID="57339a50-b218-4ba6-82dc-8d8d44702b2c" containerName="payload-pre-processing" containerID="cri-o://5d4f4f575eaf1858fa24ce4da81ca40c57ed5acab65674b5d41d6b114df6b3bb" gracePeriod=30
Sep 15 19:21:18.028715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.028677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:18.131247 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.131207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:18.158816 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.158785    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"]
Sep 15 19:21:18.168286 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.168255    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-16a967-kpk8w"]
Sep 15 19:21:18.182433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.182357    2584 generic.go:356] "Generic (PLEG): container finished" podID="57339a50-b218-4ba6-82dc-8d8d44702b2c" containerID="5d4f4f575eaf1858fa24ce4da81ca40c57ed5acab65674b5d41d6b114df6b3bb" exitCode=0
Sep 15 19:21:18.182433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.182418    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl" event={"ID":"57339a50-b218-4ba6-82dc-8d8d44702b2c","Type":"ContainerDied","Data":"5d4f4f575eaf1858fa24ce4da81ca40c57ed5acab65674b5d41d6b114df6b3bb"}
Sep 15 19:21:18.226415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.226386    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:18.234365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.234342    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:21:18.236740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.236710    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5394a355-bc69-4c72-b741-7b60fd3d278e" path="/var/lib/kubelet/pods/5394a355-bc69-4c72-b741-7b60fd3d278e/volumes"
Sep 15 19:21:18.316691 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.316651    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhqnk\" (UniqueName: \"kubernetes.io/projected/57339a50-b218-4ba6-82dc-8d8d44702b2c-kube-api-access-rhqnk\") pod \"57339a50-b218-4ba6-82dc-8d8d44702b2c\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") "
Sep 15 19:21:18.316867 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.316751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57339a50-b218-4ba6-82dc-8d8d44702b2c-plugins-config-volume\") pod \"57339a50-b218-4ba6-82dc-8d8d44702b2c\" (UID: \"57339a50-b218-4ba6-82dc-8d8d44702b2c\") "
Sep 15 19:21:18.317173 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.317144    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57339a50-b218-4ba6-82dc-8d8d44702b2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57339a50-b218-4ba6-82dc-8d8d44702b2c" (UID: "57339a50-b218-4ba6-82dc-8d8d44702b2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:18.318853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.318822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57339a50-b218-4ba6-82dc-8d8d44702b2c-kube-api-access-rhqnk" (OuterVolumeSpecName: "kube-api-access-rhqnk") pod "57339a50-b218-4ba6-82dc-8d8d44702b2c" (UID: "57339a50-b218-4ba6-82dc-8d8d44702b2c"). InnerVolumeSpecName "kube-api-access-rhqnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:18.417621 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.417571    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqnk\" (UniqueName: \"kubernetes.io/projected/57339a50-b218-4ba6-82dc-8d8d44702b2c-kube-api-access-rhqnk\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.417621 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.417602    2584 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57339a50-b218-4ba6-82dc-8d8d44702b2c-plugins-config-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:18.550926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.550873    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:18.648165 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:18.648133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:19.190117 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:19.190086    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"
Sep 15 19:21:19.190680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:19.190086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl" event={"ID":"57339a50-b218-4ba6-82dc-8d8d44702b2c","Type":"ContainerDied","Data":"8177bbe034da4e4e83e32e3956eb2437648f6ece6fffd382c14b0bf8110c4d5b"}
Sep 15 19:21:19.190680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:19.190219    2584 scope.go:117] "RemoveContainer" containerID="5d4f4f575eaf1858fa24ce4da81ca40c57ed5acab65674b5d41d6b114df6b3bb"
Sep 15 19:21:19.213930 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:19.213903    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"]
Sep 15 19:21:19.217672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:19.217638    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-16a967-6cd565498-6vkxl"]
Sep 15 19:21:19.345054 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:19.345022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:20.229818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:20.229777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:20.234998 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:20.234973    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57339a50-b218-4ba6-82dc-8d8d44702b2c" path="/var/lib/kubelet/pods/57339a50-b218-4ba6-82dc-8d8d44702b2c/volumes"
Sep 15 19:21:21.022221 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:21.022183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:21.682727 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:21.682696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:24.190928 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.190893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:24.255187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.255158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:24.502895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.502817    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"]
Sep 15 19:21:24.503038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.503011    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" containerID="cri-o://590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8" gracePeriod=30
Sep 15 19:21:24.528745 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.528702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:24.810439 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810411    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"]
Sep 15 19:21:24.810850 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810838    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy"
Sep 15 19:21:24.810911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810852    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy"
Sep 15 19:21:24.810911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810864    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5394a355-bc69-4c72-b741-7b60fd3d278e" containerName="revoke-keys"
Sep 15 19:21:24.810911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810869    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5394a355-bc69-4c72-b741-7b60fd3d278e" containerName="revoke-keys"
Sep 15 19:21:24.810911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810889    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57339a50-b218-4ba6-82dc-8d8d44702b2c" containerName="payload-pre-processing"
Sep 15 19:21:24.810911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810895    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="57339a50-b218-4ba6-82dc-8d8d44702b2c" containerName="payload-pre-processing"
Sep 15 19:21:24.811058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810962    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="57339a50-b218-4ba6-82dc-8d8d44702b2c" containerName="payload-pre-processing"
Sep 15 19:21:24.811058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810970    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5394a355-bc69-4c72-b741-7b60fd3d278e" containerName="revoke-keys"
Sep 15 19:21:24.811058 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.810978    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy"
Sep 15 19:21:24.813261 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.813245    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:24.815760 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.815739    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:21:24.815876 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.815741    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:21:24.821445 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.821418    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"]
Sep 15 19:21:24.889421 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.889386    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319a02-21bc-4b89-a10e-d54c106190bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd\" (UID: \"1e319a02-21bc-4b89-a10e-d54c106190bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:24.990524 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.990493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319a02-21bc-4b89-a10e-d54c106190bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd\" (UID: \"1e319a02-21bc-4b89-a10e-d54c106190bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:24.991126 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:24.991109    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319a02-21bc-4b89-a10e-d54c106190bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd\" (UID: \"1e319a02-21bc-4b89-a10e-d54c106190bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:25.119996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:25.119919    2584 patch_prober.go:28] interesting pod/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 15 19:21:25.119996 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:25.119986    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:21:25.125122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:25.125097    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:25.232658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:25.232601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:25.398983 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:25.398953    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"]
Sep 15 19:21:25.400417 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:21:25.400377    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e319a02_21bc_4b89_a10e_d54c106190bc.slice/crio-bfb771dccbb325c99a5fdb3b702ce4385dd77487d1e53d784ec7a8ecc500020d WatchSource:0}: Error finding container bfb771dccbb325c99a5fdb3b702ce4385dd77487d1e53d784ec7a8ecc500020d: Status 404 returned error can't find the container with id bfb771dccbb325c99a5fdb3b702ce4385dd77487d1e53d784ec7a8ecc500020d
Sep 15 19:21:26.012757 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:26.012724    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:26.242115 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:26.242087    2584 generic.go:356] "Generic (PLEG): container finished" podID="1e319a02-21bc-4b89-a10e-d54c106190bc" containerID="9c554703d12efd9c0b2eef3fc2d5260d98214f3428af1ad73277d176b6550b9f" exitCode=0
Sep 15 19:21:26.242472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:26.242166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd" event={"ID":"1e319a02-21bc-4b89-a10e-d54c106190bc","Type":"ContainerDied","Data":"9c554703d12efd9c0b2eef3fc2d5260d98214f3428af1ad73277d176b6550b9f"}
Sep 15 19:21:26.242472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:26.242209    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd" event={"ID":"1e319a02-21bc-4b89-a10e-d54c106190bc","Type":"ContainerStarted","Data":"bfb771dccbb325c99a5fdb3b702ce4385dd77487d1e53d784ec7a8ecc500020d"}
Sep 15 19:21:26.822396 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:26.822363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:27.384488 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:27.384461    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:27.512024 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:27.511989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:27.519062 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:27.519032    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319a02-21bc-4b89-a10e-d54c106190bc-maas-api-service-ca\") pod \"1e319a02-21bc-4b89-a10e-d54c106190bc\" (UID: \"1e319a02-21bc-4b89-a10e-d54c106190bc\") "
Sep 15 19:21:27.519411 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:27.519389    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e319a02-21bc-4b89-a10e-d54c106190bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e319a02-21bc-4b89-a10e-d54c106190bc" (UID: "1e319a02-21bc-4b89-a10e-d54c106190bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:27.620101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:27.620051    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e319a02-21bc-4b89-a10e-d54c106190bc-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:28.239127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:28.239081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:28.255728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:28.255700    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"
Sep 15 19:21:28.255836 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:28.255734    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd" event={"ID":"1e319a02-21bc-4b89-a10e-d54c106190bc","Type":"ContainerDied","Data":"bfb771dccbb325c99a5fdb3b702ce4385dd77487d1e53d784ec7a8ecc500020d"}
Sep 15 19:21:28.255836 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:28.255777    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfb771dccbb325c99a5fdb3b702ce4385dd77487d1e53d784ec7a8ecc500020d"
Sep 15 19:21:28.857987 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:28.857943    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:28.958771 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:28.958738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:29.759934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.759906    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:21:29.843452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843384    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-token\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843454    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-envoy\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843484    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-data\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843517    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istiod-ca-cert\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843604    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-podinfo\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843664    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lplft\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-kube-api-access-lplft\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843691    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-socket\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843760    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-credential-socket\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843790    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-certs\") pod \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\" (UID: \"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6\") "
Sep 15 19:21:29.843897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.843822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:29.844092 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844056    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:29.844157 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844085    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:29.844279 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844250    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:29.844279 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844266    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.844458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:29.844458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844287    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.844458 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.844320    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.845813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.845784    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:29.845813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.845795    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:29.845976 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.845785    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:21:29.845976 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.845870    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-kube-api-access-lplft" (OuterVolumeSpecName: "kube-api-access-lplft") pod "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" (UID: "32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6"). InnerVolumeSpecName "kube-api-access-lplft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:29.945927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.945896    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.945927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.945929    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.946294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.945939    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.946294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.945948    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lplft\" (UniqueName: \"kubernetes.io/projected/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-kube-api-access-lplft\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.946294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.945958    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:29.946294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:29.945967    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:30.182896 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.182804    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9002-22a437"]
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184042    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184093    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184146    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e319a02-21bc-4b89-a10e-d54c106190bc" containerName="revoke-keys"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184156    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e319a02-21bc-4b89-a10e-d54c106190bc" containerName="revoke-keys"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184170    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184195    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184218    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184226    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184338    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e319a02-21bc-4b89-a10e-d54c106190bc" containerName="revoke-keys"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184354    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184363    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerName="istio-proxy"
Sep 15 19:21:30.184645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.184372    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ceefe71-9eda-484a-9d5a-6b629816e161" containerName="istio-proxy"
Sep 15 19:21:30.186864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.186842    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9002-22a437"]
Sep 15 19:21:30.187005 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.186931    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:30.189496 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.189476    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pgkcn\""
Sep 15 19:21:30.270528 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.270502    2584 generic.go:356] "Generic (PLEG): container finished" podID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" containerID="590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8" exitCode=0
Sep 15 19:21:30.270645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.270555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"
Sep 15 19:21:30.270645 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.270585    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" event={"ID":"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6","Type":"ContainerDied","Data":"590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8"}
Sep 15 19:21:30.270755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.270648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn" event={"ID":"32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6","Type":"ContainerDied","Data":"0d823c772522a07c81e6d603c8b1d9fc02389c1931edd73ab25969aab1c54b4b"}
Sep 15 19:21:30.270755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.270681    2584 scope.go:117] "RemoveContainer" containerID="590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8"
Sep 15 19:21:30.280878 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.280859    2584 scope.go:117] "RemoveContainer" containerID="e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"
Sep 15 19:21:30.287053 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.287030    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"]
Sep 15 19:21:30.289878 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.289862    2584 scope.go:117] "RemoveContainer" containerID="590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8"
Sep 15 19:21:30.290267 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:30.290240    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8\": container with ID starting with 590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8 not found: ID does not exist" containerID="590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8"
Sep 15 19:21:30.290478 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.290452    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8"} err="failed to get container status \"590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8\": rpc error: code = NotFound desc = could not find container \"590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8\": container with ID starting with 590426b9bfb8ed07a07becd8e078b46b54f16d5399d14cffd678fabaac16afa8 not found: ID does not exist"
Sep 15 19:21:30.290550 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.290483    2584 scope.go:117] "RemoveContainer" containerID="e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"
Sep 15 19:21:30.290816 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:30.290797    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63\": container with ID starting with e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63 not found: ID does not exist" containerID="e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"
Sep 15 19:21:30.290881 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.290822    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63"} err="failed to get container status \"e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63\": rpc error: code = NotFound desc = could not find container \"e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63\": container with ID starting with e73e6d329352e4c37e381f637e6f3ef2f49c61688a09e3e35bc5abd5ad0ccc63 not found: ID does not exist"
Sep 15 19:21:30.293304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.293285    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-16a967-openshift-default-84f8fb575d-7k8kn"]
Sep 15 19:21:30.349516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.349485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqpkt\" (UniqueName: \"kubernetes.io/projected/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2-kube-api-access-gqpkt\") pod \"test-curl-9002-22a437\" (UID: \"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2\") " pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:30.451128 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.451046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqpkt\" (UniqueName: \"kubernetes.io/projected/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2-kube-api-access-gqpkt\") pod \"test-curl-9002-22a437\" (UID: \"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2\") " pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:30.458410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.458389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqpkt\" (UniqueName: \"kubernetes.io/projected/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2-kube-api-access-gqpkt\") pod \"test-curl-9002-22a437\" (UID: \"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2\") " pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:30.497420 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.497387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:30.767129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.767099    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9002-22a437"]
Sep 15 19:21:30.768139 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:21:30.768112    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd54660fe_5c80_48ee_b382_0a3a9cdbb5f2.slice/crio-6b43b07f081ee4d2c5afebed87f217cb7a6ef288540464e8b483e38933595e08 WatchSource:0}: Error finding container 6b43b07f081ee4d2c5afebed87f217cb7a6ef288540464e8b483e38933595e08: Status 404 returned error can't find the container with id 6b43b07f081ee4d2c5afebed87f217cb7a6ef288540464e8b483e38933595e08
Sep 15 19:21:30.770065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:30.770045    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:21:31.279388 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:31.279354    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9002-22a437" event={"ID":"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2","Type":"ContainerStarted","Data":"6b43b07f081ee4d2c5afebed87f217cb7a6ef288540464e8b483e38933595e08"}
Sep 15 19:21:32.234858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:32.234829    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6" path="/var/lib/kubelet/pods/32cbf4e1-2bce-47a3-8aaa-d3fd82eb83c6/volumes"
Sep 15 19:21:32.287362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:32.287331    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9002-22a437" event={"ID":"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2","Type":"ContainerStarted","Data":"616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4"}
Sep 15 19:21:32.304478 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:32.304435    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9002-22a437" podStartSLOduration=1.427118814 podStartE2EDuration="2.304420158s" totalImagesPullingTime="877.301344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:21:32.302922685 +0000 UTC m=+1844.457291154" watchObservedRunningTime="2026-09-15 19:21:32.304420158 +0000 UTC m=+1844.458788639"
Sep 15 19:21:32.871299 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:32.871269    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9002-22a437"]
Sep 15 19:21:32.874212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:32.874182    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9002-22a437"]
Sep 15 19:21:33.294516 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:33.294489    2584 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9002-22a437" container="test-curl-9002-22a437"
Sep 15 19:21:34.163525 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:34.163491    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"]
Sep 15 19:21:34.182882 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:34.182851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:34.300926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:34.300858    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9002-22a437" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" containerName="test-curl-9002-22a437" containerID="cri-o://616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4" gracePeriod=2
Sep 15 19:21:35.181908 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.181875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:35.385524 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.385497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:35.391361 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.391333    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"]
Sep 15 19:21:35.394073 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.394047    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-362ea4-29824995-snh27"]
Sep 15 19:21:35.456761 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.456673    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:35.521816 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.521777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:35.575590 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.575555    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"]
Sep 15 19:21:35.578442 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.578421    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-362ea4-d6pbd"]
Sep 15 19:21:35.585122 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.585098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:35.662166 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.662118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:35.922677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:35.922602    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:36.017946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.017902    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:36.235074 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.234999    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e319a02-21bc-4b89-a10e-d54c106190bc" path="/var/lib/kubelet/pods/1e319a02-21bc-4b89-a10e-d54c106190bc/volumes"
Sep 15 19:21:36.235307 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.235294    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3bf24d4-8d25-4674-afdc-aab1e1707a5e" path="/var/lib/kubelet/pods/a3bf24d4-8d25-4674-afdc-aab1e1707a5e/volumes"
Sep 15 19:21:36.484287 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.484265    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:36.515256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.515224    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqpkt\" (UniqueName: \"kubernetes.io/projected/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2-kube-api-access-gqpkt\") pod \"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2\" (UID: \"d54660fe-5c80-48ee-b382-0a3a9cdbb5f2\") "
Sep 15 19:21:36.517263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.517239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2-kube-api-access-gqpkt" (OuterVolumeSpecName: "kube-api-access-gqpkt") pod "d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" (UID: "d54660fe-5c80-48ee-b382-0a3a9cdbb5f2"). InnerVolumeSpecName "kube-api-access-gqpkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:36.616957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.616915    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqpkt\" (UniqueName: \"kubernetes.io/projected/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2-kube-api-access-gqpkt\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:36.842693 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:36.842600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:37.326212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:37.326172    2584 generic.go:356] "Generic (PLEG): container finished" podID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" containerID="616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4" exitCode=137
Sep 15 19:21:37.326212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:37.326212    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:37.326462 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:37.326259    2584 scope.go:117] "RemoveContainer" containerID="616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4"
Sep 15 19:21:37.336366 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:37.336348    2584 scope.go:117] "RemoveContainer" containerID="616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4"
Sep 15 19:21:37.336501 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:37.336477    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9002-22a437\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:37.336658 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:37.336635    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4\": container with ID starting with 616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4 not found: ID does not exist" containerID="616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4"
Sep 15 19:21:37.336724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:37.336672    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4"} err="failed to get container status \"616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4\": rpc error: code = NotFound desc = could not find container \"616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4\": container with ID starting with 616bc559f1988450c0cb318acf851ad672b9179cf8af6a55917deb2b975a65f4 not found: ID does not exist"
Sep 15 19:21:37.617045 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:37.616938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:38.233558 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:38.233521    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9002-22a437\" is forbidden: User \"system:node:ip-10-0-143-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-252.ec2.internal' and this object" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" pod="opendatahub/test-curl-9002-22a437"
Sep 15 19:21:38.236070 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:38.236044    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" path="/var/lib/kubelet/pods/d54660fe-5c80-48ee-b382-0a3a9cdbb5f2/volumes"
Sep 15 19:21:38.485490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:38.485373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:39.654974 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:39.654946    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:41.085747 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.085712    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"]
Sep 15 19:21:41.086365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.086346    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" containerName="test-curl-9002-22a437"
Sep 15 19:21:41.086365 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.086367    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" containerName="test-curl-9002-22a437"
Sep 15 19:21:41.086487 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.086471    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54660fe-5c80-48ee-b382-0a3a9cdbb5f2" containerName="test-curl-9002-22a437"
Sep 15 19:21:41.089206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.089185    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.091603 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.091582    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c6d10a\"/\"sub-test-model-c6d10a-kserve-self-signed-certs\""
Sep 15 19:21:41.097956 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.097935    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"]
Sep 15 19:21:41.111598 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.111576    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:41.162088 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.162057    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-home\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.162243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.162094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-dshm\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.162243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.162120    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjd4\" (UniqueName: \"kubernetes.io/projected/e6b6e0be-d775-450d-b781-27d153aa722c-kube-api-access-lfjd4\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.162243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.162216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-model-cache\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.162355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.162270    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-tmp-dir\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.162355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.162306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b6e0be-d775-450d-b781-27d153aa722c-tls-certs\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.262979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.262940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-tmp-dir\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.262990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b6e0be-d775-450d-b781-27d153aa722c-tls-certs\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263247    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-home\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263333 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-dshm\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263392 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjd4\" (UniqueName: \"kubernetes.io/projected/e6b6e0be-d775-450d-b781-27d153aa722c-kube-api-access-lfjd4\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263392 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-tmp-dir\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263502 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263447    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-model-cache\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-home\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.263821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.263803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-model-cache\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.265484 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.265465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-dshm\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.265844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.265825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b6e0be-d775-450d-b781-27d153aa722c-tls-certs\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.271110 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.271088    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjd4\" (UniqueName: \"kubernetes.io/projected/e6b6e0be-d775-450d-b781-27d153aa722c-kube-api-access-lfjd4\") pod \"sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") " pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.399362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.399286    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:41.670756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.670731    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"]
Sep 15 19:21:41.672679 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:21:41.672650    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6b6e0be_d775_450d_b781_27d153aa722c.slice/crio-d201b8ecf79aeeb09a3bd957f1752af31e686c42cdc19fbf08a1750127c746e9 WatchSource:0}: Error finding container d201b8ecf79aeeb09a3bd957f1752af31e686c42cdc19fbf08a1750127c746e9: Status 404 returned error can't find the container with id d201b8ecf79aeeb09a3bd957f1752af31e686c42cdc19fbf08a1750127c746e9
Sep 15 19:21:41.749557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.749526    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:41.817472 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.817442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:41.967287 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.967204    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"]
Sep 15 19:21:41.967452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.967406    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy" containerID="cri-o://77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e" gracePeriod=30
Sep 15 19:21:41.990243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:41.990213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:42.308542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.308509    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"]
Sep 15 19:21:42.311120 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.311104    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:42.313505 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.313482    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:21:42.313667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.313503    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:21:42.318962 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.318933    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"]
Sep 15 19:21:42.375638 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.375334    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2\" (UID: \"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:42.377273 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.377224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz" event={"ID":"e6b6e0be-d775-450d-b781-27d153aa722c","Type":"ContainerStarted","Data":"91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968"}
Sep 15 19:21:42.377403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.377293    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz" event={"ID":"e6b6e0be-d775-450d-b781-27d153aa722c","Type":"ContainerStarted","Data":"d201b8ecf79aeeb09a3bd957f1752af31e686c42cdc19fbf08a1750127c746e9"}
Sep 15 19:21:42.397690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.397625    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz" podStartSLOduration=1.3975899520000001 podStartE2EDuration="1.397589952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:21:42.393405893 +0000 UTC m=+1854.547774383" watchObservedRunningTime="2026-09-15 19:21:42.397589952 +0000 UTC m=+1854.551958421"
Sep 15 19:21:42.477014 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.476982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2\" (UID: \"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:42.477635 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.477592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2\" (UID: \"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:42.621809 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.621696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:42.624528 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.624498    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:42.898027 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:42.897990    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"]
Sep 15 19:21:42.899411 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:21:42.899385    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22c8f49_69e8_4c5e_ac67_9a3cc23d0b3a.slice/crio-e42a89ca14898c5d7951e0b54b2e082606b71b7b916495a89e2b83127edf138c WatchSource:0}: Error finding container e42a89ca14898c5d7951e0b54b2e082606b71b7b916495a89e2b83127edf138c: Status 404 returned error can't find the container with id e42a89ca14898c5d7951e0b54b2e082606b71b7b916495a89e2b83127edf138c
Sep 15 19:21:43.385889 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:43.385851    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2" event={"ID":"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a","Type":"ContainerStarted","Data":"a005796b88777c7a9820ead4c7b65e4abc2b942bb8113d28cae0b207febdd979"}
Sep 15 19:21:43.386329 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:43.385905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2" event={"ID":"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a","Type":"ContainerStarted","Data":"e42a89ca14898c5d7951e0b54b2e082606b71b7b916495a89e2b83127edf138c"}
Sep 15 19:21:43.391386 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:43.391359    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:43.401853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:43.401813    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2" podStartSLOduration=1.40179899 podStartE2EDuration="1.40179899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:21:43.400181985 +0000 UTC m=+1855.554550454" watchObservedRunningTime="2026-09-15 19:21:43.40179899 +0000 UTC m=+1855.556167458"
Sep 15 19:21:44.028946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:44.028904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:44.394667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:44.394545    2584 generic.go:356] "Generic (PLEG): container finished" podID="a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a" containerID="a005796b88777c7a9820ead4c7b65e4abc2b942bb8113d28cae0b207febdd979" exitCode=0
Sep 15 19:21:44.394667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:44.394641    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2" event={"ID":"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a","Type":"ContainerDied","Data":"a005796b88777c7a9820ead4c7b65e4abc2b942bb8113d28cae0b207febdd979"}
Sep 15 19:21:44.660453 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:44.660371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:45.211087 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:45.211043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:45.544197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:45.544176    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:45.609558 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:45.609527    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a-maas-api-service-ca\") pod \"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a\" (UID: \"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a\") "
Sep 15 19:21:45.609883 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:45.609860    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a" (UID: "a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:45.710422 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:45.710386    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:46.412972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:46.412934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2" event={"ID":"a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a","Type":"ContainerDied","Data":"e42a89ca14898c5d7951e0b54b2e082606b71b7b916495a89e2b83127edf138c"}
Sep 15 19:21:46.412972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:46.412973    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"
Sep 15 19:21:46.412972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:46.412982    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42a89ca14898c5d7951e0b54b2e082606b71b7b916495a89e2b83127edf138c"
Sep 15 19:21:47.216761 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.216736    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:21:47.325368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325337    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istiod-ca-cert\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325547 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325380    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-token\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325547 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325411    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-socket\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325697 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325541    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-certs\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325697 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325665    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-podinfo\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325697 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:47.325845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325712    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-envoy\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325769    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-data\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.325845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325792    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:47.325845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325815    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:21:47.326132 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325854    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-credential-socket\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.326132 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.325885    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58zgf\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-kube-api-access-58zgf\") pod \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\" (UID: \"fdb4cb51-3ed1-43bb-b281-c35dfb310492\") "
Sep 15 19:21:47.326906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.326221    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:47.326906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.326276    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.326906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.326294    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.326906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.326308    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.326906 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.326277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-data" (OuterVolumeSpecName: "istio-data") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:47.328160 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.328121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:47.328454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.328383    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:47.328454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.328417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:21:47.328586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.328482    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-kube-api-access-58zgf" (OuterVolumeSpecName: "kube-api-access-58zgf") pod "fdb4cb51-3ed1-43bb-b281-c35dfb310492" (UID: "fdb4cb51-3ed1-43bb-b281-c35dfb310492"). InnerVolumeSpecName "kube-api-access-58zgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:47.421351 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.421320    2584 generic.go:356] "Generic (PLEG): container finished" podID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerID="77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e" exitCode=0
Sep 15 19:21:47.421533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.421381    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"
Sep 15 19:21:47.421533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.421405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" event={"ID":"fdb4cb51-3ed1-43bb-b281-c35dfb310492","Type":"ContainerDied","Data":"77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e"}
Sep 15 19:21:47.421533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.421462    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6" event={"ID":"fdb4cb51-3ed1-43bb-b281-c35dfb310492","Type":"ContainerDied","Data":"acb90759c7e816fb10ae65681e3b172b392f5c00618296063b8bc70be506321c"}
Sep 15 19:21:47.421533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.421484    2584 scope.go:117] "RemoveContainer" containerID="77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e"
Sep 15 19:21:47.427135 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.427110    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.427256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.427140    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.427256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.427155    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.427256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.427171    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdb4cb51-3ed1-43bb-b281-c35dfb310492-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.427256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.427185    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zgf\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-kube-api-access-58zgf\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.427256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.427200    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdb4cb51-3ed1-43bb-b281-c35dfb310492-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:47.432244 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.432226    2584 scope.go:117] "RemoveContainer" containerID="39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"
Sep 15 19:21:47.441412 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.441394    2584 scope.go:117] "RemoveContainer" containerID="77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e"
Sep 15 19:21:47.441735 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:47.441713    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e\": container with ID starting with 77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e not found: ID does not exist" containerID="77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e"
Sep 15 19:21:47.441834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.441745    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e"} err="failed to get container status \"77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e\": rpc error: code = NotFound desc = could not find container \"77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e\": container with ID starting with 77675ca3a15cedf809df31497799d97a9b22b385dccb330657c7f109c03e134e not found: ID does not exist"
Sep 15 19:21:47.441834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.441771    2584 scope.go:117] "RemoveContainer" containerID="39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"
Sep 15 19:21:47.442095 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:21:47.442072    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60\": container with ID starting with 39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60 not found: ID does not exist" containerID="39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"
Sep 15 19:21:47.442207 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.442185    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60"} err="failed to get container status \"39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60\": rpc error: code = NotFound desc = could not find container \"39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60\": container with ID starting with 39af2e40012e2261348db1c79c1f1dd4b08d8e3c833d415035caf00c0ffeeb60 not found: ID does not exist"
Sep 15 19:21:47.445374 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.445355    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"]
Sep 15 19:21:47.452731 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:47.452713    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-362ea4-openshift-default-fcbbf5fb6-bcxw6"]
Sep 15 19:21:48.239525 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:48.239491    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" path="/var/lib/kubelet/pods/fdb4cb51-3ed1-43bb-b281-c35dfb310492/volumes"
Sep 15 19:21:48.690417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:48.690386    2584 scope.go:117] "RemoveContainer" containerID="401f3781525c565977280dc19154211d34795f816a0b3617227dc375954c5936"
Sep 15 19:21:48.699720 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:48.699701    2584 scope.go:117] "RemoveContainer" containerID="821773835e9daac1136bca68324ab575c0f8b783f2a38512e9b36b4fdc4c3c9f"
Sep 15 19:21:48.708346 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:48.708324    2584 scope.go:117] "RemoveContainer" containerID="da4f49f6f78dc6f94406b7f0fada2ba724cd013120a3ac9a44e1b8ca89a4becd"
Sep 15 19:21:48.716251 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:48.716231    2584 scope.go:117] "RemoveContainer" containerID="1ff83c140b14b65a18388b86532977e4249a2574b014451bf3c2522a4540632d"
Sep 15 19:21:48.724427 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:48.724406    2584 scope.go:117] "RemoveContainer" containerID="c8ffb27125f5a0b7b41c96b1b5b7f900db544a0a29ad1f4f7718b7ab948fe505"
Sep 15 19:21:49.187708 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.187671    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q" podUID="8809b592-d109-4781-9224-26be35de469d" containerName="main" containerID="cri-o://4f287b9f7140301bfca2174b477fe8acc1075d2ae871251c2eec96909a9abaf6" gracePeriod=45
Sep 15 19:21:49.438785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.438701    2584 generic.go:356] "Generic (PLEG): container finished" podID="8809b592-d109-4781-9224-26be35de469d" containerID="4f287b9f7140301bfca2174b477fe8acc1075d2ae871251c2eec96909a9abaf6" exitCode=0
Sep 15 19:21:49.439104 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.438786    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q" event={"ID":"8809b592-d109-4781-9224-26be35de469d","Type":"ContainerDied","Data":"4f287b9f7140301bfca2174b477fe8acc1075d2ae871251c2eec96909a9abaf6"}
Sep 15 19:21:49.439104 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.438832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q" event={"ID":"8809b592-d109-4781-9224-26be35de469d","Type":"ContainerDied","Data":"274d991046a757916bf44953e5651e32d593ca0109806a10b0982137b9ac2eb9"}
Sep 15 19:21:49.439104 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.438849    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="274d991046a757916bf44953e5651e32d593ca0109806a10b0982137b9ac2eb9"
Sep 15 19:21:49.444729 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.444709    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:21:49.551236 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551203    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-tmp-dir\") pod \"8809b592-d109-4781-9224-26be35de469d\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") "
Sep 15 19:21:49.551509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-model-cache\") pod \"8809b592-d109-4781-9224-26be35de469d\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") "
Sep 15 19:21:49.551509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551276    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kln5g\" (UniqueName: \"kubernetes.io/projected/8809b592-d109-4781-9224-26be35de469d-kube-api-access-kln5g\") pod \"8809b592-d109-4781-9224-26be35de469d\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") "
Sep 15 19:21:49.551509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8809b592-d109-4781-9224-26be35de469d-tls-certs\") pod \"8809b592-d109-4781-9224-26be35de469d\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") "
Sep 15 19:21:49.551509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551362    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-home\") pod \"8809b592-d109-4781-9224-26be35de469d\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") "
Sep 15 19:21:49.551509 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551443    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-dshm\") pod \"8809b592-d109-4781-9224-26be35de469d\" (UID: \"8809b592-d109-4781-9224-26be35de469d\") "
Sep 15 19:21:49.551808 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551551    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8809b592-d109-4781-9224-26be35de469d" (UID: "8809b592-d109-4781-9224-26be35de469d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:49.551808 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551593    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-home" (OuterVolumeSpecName: "home") pod "8809b592-d109-4781-9224-26be35de469d" (UID: "8809b592-d109-4781-9224-26be35de469d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:49.551808 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551630    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8809b592-d109-4781-9224-26be35de469d" (UID: "8809b592-d109-4781-9224-26be35de469d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:49.551918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551872    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.551918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551887    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.551918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.551895    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.553660 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.553637    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8809b592-d109-4781-9224-26be35de469d-kube-api-access-kln5g" (OuterVolumeSpecName: "kube-api-access-kln5g") pod "8809b592-d109-4781-9224-26be35de469d" (UID: "8809b592-d109-4781-9224-26be35de469d"). InnerVolumeSpecName "kube-api-access-kln5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:21:49.553919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.553902    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-dshm" (OuterVolumeSpecName: "dshm") pod "8809b592-d109-4781-9224-26be35de469d" (UID: "8809b592-d109-4781-9224-26be35de469d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:21:49.553986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.553905    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8809b592-d109-4781-9224-26be35de469d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8809b592-d109-4781-9224-26be35de469d" (UID: "8809b592-d109-4781-9224-26be35de469d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:21:49.652755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.652721    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8809b592-d109-4781-9224-26be35de469d-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.652755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.652747    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8809b592-d109-4781-9224-26be35de469d-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:49.652755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:49.652757    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kln5g\" (UniqueName: \"kubernetes.io/projected/8809b592-d109-4781-9224-26be35de469d-kube-api-access-kln5g\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:21:50.445479 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:50.445447    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"
Sep 15 19:21:50.461991 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:50.461962    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"]
Sep 15 19:21:50.473113 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:50.473084    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/auth-test-model-c6d10a-kserve-6d8bdcf47-s4c6q"]
Sep 15 19:21:51.399959 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:51.399933    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:51.400129 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:51.399973    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:51.412630 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:51.412582    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:51.462858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:51.462825    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:21:52.233894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:52.233865    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8809b592-d109-4781-9224-26be35de469d" path="/var/lib/kubelet/pods/8809b592-d109-4781-9224-26be35de469d/volumes"
Sep 15 19:21:52.980831 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:52.980794    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:52.991586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:52.991552    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"]
Sep 15 19:21:52.994452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:52.994430    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1b0e3f-29824995-gkpph"]
Sep 15 19:21:53.079918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.079887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:53.180265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.180238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:53.187149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.187125    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"]
Sep 15 19:21:53.189598 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.189579    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1b0e3f-xjmf2"]
Sep 15 19:21:53.241014 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.240944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:53.545626 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.545590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:53.658116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:53.658084    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:54.236194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:54.236160    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7effa0-6bcd-4ebf-b478-35c2af84f83e" path="/var/lib/kubelet/pods/0a7effa0-6bcd-4ebf-b478-35c2af84f83e/volumes"
Sep 15 19:21:54.236582 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:54.236466    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a" path="/var/lib/kubelet/pods/a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a/volumes"
Sep 15 19:21:54.287707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:54.287673    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:55.026725 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:55.026693    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:59.370993 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:59.370961    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:59.450343 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:59.450288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:21:59.648660 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:59.648557    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"]
Sep 15 19:21:59.648914 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:59.648885    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy" containerID="cri-o://e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40" gracePeriod=30
Sep 15 19:21:59.707799 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:21:59.707770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:22:00.465167 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:00.465141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:22:04.904023 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:04.903994    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:22:05.013723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013685    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istiod-ca-cert\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.013904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013781    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-credential-socket\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.013904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013805    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-envoy\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.013904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013861    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snwkk\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-kube-api-access-snwkk\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.013904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013888    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-certs\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.014134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013920    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-data\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.014134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013951    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-podinfo\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.014134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.013981    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-socket\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.014134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014011    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-token\") pod \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\" (UID: \"fe4f654e-81b8-4480-a34c-cdb51f2831d3\") "
Sep 15 19:22:05.014134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014084    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:22:05.014134 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014105    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:05.014533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:05.014533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014315    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:05.014533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014336    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:05.014533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014497    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.014533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014518    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.014533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014533    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.014893 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014547    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.014893 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.014560    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.016177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.016151    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:22:05.016289 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.016254    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:05.016358 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.016323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-kube-api-access-snwkk" (OuterVolumeSpecName: "kube-api-access-snwkk") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "kube-api-access-snwkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:22:05.016358 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.016337    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe4f654e-81b8-4480-a34c-cdb51f2831d3" (UID: "fe4f654e-81b8-4480-a34c-cdb51f2831d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:22:05.115109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.115062    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.115109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.115103    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.115109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.115113    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4f654e-81b8-4480-a34c-cdb51f2831d3-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.115109 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.115122    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snwkk\" (UniqueName: \"kubernetes.io/projected/fe4f654e-81b8-4480-a34c-cdb51f2831d3-kube-api-access-snwkk\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:22:05.543969 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.543932    2584 generic.go:356] "Generic (PLEG): container finished" podID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerID="e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40" exitCode=0
Sep 15 19:22:05.544152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.543984    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"
Sep 15 19:22:05.544152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.544017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" event={"ID":"fe4f654e-81b8-4480-a34c-cdb51f2831d3","Type":"ContainerDied","Data":"e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40"}
Sep 15 19:22:05.544152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.544073    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx" event={"ID":"fe4f654e-81b8-4480-a34c-cdb51f2831d3","Type":"ContainerDied","Data":"26fccc71c5f12af82120aab4fb7458d4064e34b775e8b4e51524c13a5f3ef8e7"}
Sep 15 19:22:05.544152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.544097    2584 scope.go:117] "RemoveContainer" containerID="e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40"
Sep 15 19:22:05.554917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.554891    2584 scope.go:117] "RemoveContainer" containerID="1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"
Sep 15 19:22:05.564603 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.564586    2584 scope.go:117] "RemoveContainer" containerID="e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40"
Sep 15 19:22:05.564924 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:22:05.564902    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40\": container with ID starting with e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40 not found: ID does not exist" containerID="e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40"
Sep 15 19:22:05.565012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.564933    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40"} err="failed to get container status \"e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40\": rpc error: code = NotFound desc = could not find container \"e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40\": container with ID starting with e828a35b17e01d3c7b6a2e50282de0a73a494da3f961374c9912bc438e38da40 not found: ID does not exist"
Sep 15 19:22:05.565012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.564951    2584 scope.go:117] "RemoveContainer" containerID="1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"
Sep 15 19:22:05.565250 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:22:05.565234    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d\": container with ID starting with 1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d not found: ID does not exist" containerID="1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"
Sep 15 19:22:05.565300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.565254    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d"} err="failed to get container status \"1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d\": rpc error: code = NotFound desc = could not find container \"1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d\": container with ID starting with 1dc9b4fbcf5f586454ebdb7f6cf1bdb5d0f5844129229517b2aceed43c44884d not found: ID does not exist"
Sep 15 19:22:05.567250 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.567230    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"]
Sep 15 19:22:05.569810 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:05.569791    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1b0e3f-openshift-default-56458c4456-v4nvx"]
Sep 15 19:22:06.235238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.235203    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" path="/var/lib/kubelet/pods/fe4f654e-81b8-4480-a34c-cdb51f2831d3/volumes"
Sep 15 19:22:06.482353 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482322    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"]
Sep 15 19:22:06.482751 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482737    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy"
Sep 15 19:22:06.482805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482753    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy"
Sep 15 19:22:06.482805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482771    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy"
Sep 15 19:22:06.482805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482777    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy"
Sep 15 19:22:06.482805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482790    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8809b592-d109-4781-9224-26be35de469d" containerName="main"
Sep 15 19:22:06.482805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482796    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8809b592-d109-4781-9224-26be35de469d" containerName="main"
Sep 15 19:22:06.482805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482802    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482808    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482820    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a" containerName="revoke-keys"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482825    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a" containerName="revoke-keys"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482881    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482891    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482897    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdb4cb51-3ed1-43bb-b281-c35dfb310492" containerName="istio-proxy"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482903    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8809b592-d109-4781-9224-26be35de469d" containerName="main"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482910    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22c8f49-69e8-4c5e-ac67-9a3cc23d0b3a" containerName="revoke-keys"
Sep 15 19:22:06.483000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.482916    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy"
Sep 15 19:22:06.487473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.487427    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.489900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.489877    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"kube-root-ca.crt\""
Sep 15 19:22:06.490810 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.490788    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"openshift-service-ca.crt\""
Sep 15 19:22:06.490946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.490865    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"sub-test-model-415f87-kserve-self-signed-certs\""
Sep 15 19:22:06.490946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.490895    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"default-dockercfg-j4qsp\""
Sep 15 19:22:06.495643 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.495601    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"]
Sep 15 19:22:06.527295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.527273    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tmp-dir\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.527400 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.527355    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-dshm\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.527400 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.527380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7fm\" (UniqueName: \"kubernetes.io/projected/9d436bbe-0cd8-4149-8870-3cecfb0aff24-kube-api-access-vj7fm\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.527493 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.527429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tls-certs\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.527493 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.527472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-home\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.527574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.527526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-model-cache\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.585941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.585910    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:22:06.628445 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-dshm\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.628585 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7fm\" (UniqueName: \"kubernetes.io/projected/9d436bbe-0cd8-4149-8870-3cecfb0aff24-kube-api-access-vj7fm\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.628585 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tls-certs\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.628585 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-home\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.628798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-model-cache\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.628798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628666    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tmp-dir\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.628964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-home\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.629030 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.628995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-model-cache\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.629083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.629031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tmp-dir\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.630649 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.630599    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-dshm\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.630980 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.630963    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tls-certs\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.636042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.636016    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7fm\" (UniqueName: \"kubernetes.io/projected/9d436bbe-0cd8-4149-8870-3cecfb0aff24-kube-api-access-vj7fm\") pod \"sub-test-model-415f87-kserve-5bf9865885-4blkx\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") " pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:06.797636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:06.797585    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:07.072815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:07.072787    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"]
Sep 15 19:22:07.073791 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:22:07.073766    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d436bbe_0cd8_4149_8870_3cecfb0aff24.slice/crio-76b8b8e876c0a31a88ea91f58b9a79b8f596a86d05751e3d985ec58d3e9ea806 WatchSource:0}: Error finding container 76b8b8e876c0a31a88ea91f58b9a79b8f596a86d05751e3d985ec58d3e9ea806: Status 404 returned error can't find the container with id 76b8b8e876c0a31a88ea91f58b9a79b8f596a86d05751e3d985ec58d3e9ea806
Sep 15 19:22:07.561118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:07.561085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx" event={"ID":"9d436bbe-0cd8-4149-8870-3cecfb0aff24","Type":"ContainerStarted","Data":"d478c62e2f2c98fd1063d64d61cf11705f9ece840d7677b15201d752bc4ecf76"}
Sep 15 19:22:07.561118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:07.561125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx" event={"ID":"9d436bbe-0cd8-4149-8870-3cecfb0aff24","Type":"ContainerStarted","Data":"76b8b8e876c0a31a88ea91f58b9a79b8f596a86d05751e3d985ec58d3e9ea806"}
Sep 15 19:22:07.578294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:07.578243    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx" podStartSLOduration=1.578228819 podStartE2EDuration="1.578228819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:22:07.576795096 +0000 UTC m=+1879.731163564" watchObservedRunningTime="2026-09-15 19:22:07.578228819 +0000 UTC m=+1879.732597288"
Sep 15 19:22:08.926483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:08.926448    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:22:16.797913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:16.797873    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:16.798378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:16.797935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:16.810279 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:16.810252    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:17.635824 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:17.635787    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:22:48.789373 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:48.789341    2584 scope.go:117] "RemoveContainer" containerID="554e7795b127ad285f663ac101508e59a5f84885f618c4c89a452b384f88751b"
Sep 15 19:22:48.823475 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:48.823444    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"]
Sep 15 19:22:48.836022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:48.835994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:22:49.796046 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:49.796015    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"]
Sep 15 19:22:49.812744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:49.812718    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:22:53.200723 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:22:53.200693    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:03.844499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:03.844452    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz" podUID="e6b6e0be-d775-450d-b781-27d153aa722c" containerName="main" containerID="cri-o://91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968" gracePeriod=45
Sep 15 19:23:04.102344 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.102277    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:23:04.202853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.202821    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-dshm\") pod \"e6b6e0be-d775-450d-b781-27d153aa722c\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") "
Sep 15 19:23:04.202853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.202861    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfjd4\" (UniqueName: \"kubernetes.io/projected/e6b6e0be-d775-450d-b781-27d153aa722c-kube-api-access-lfjd4\") pod \"e6b6e0be-d775-450d-b781-27d153aa722c\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") "
Sep 15 19:23:04.203086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.202902    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-tmp-dir\") pod \"e6b6e0be-d775-450d-b781-27d153aa722c\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") "
Sep 15 19:23:04.203086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.202934    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-model-cache\") pod \"e6b6e0be-d775-450d-b781-27d153aa722c\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") "
Sep 15 19:23:04.203086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203036    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-home\") pod \"e6b6e0be-d775-450d-b781-27d153aa722c\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") "
Sep 15 19:23:04.203086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203068    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b6e0be-d775-450d-b781-27d153aa722c-tls-certs\") pod \"e6b6e0be-d775-450d-b781-27d153aa722c\" (UID: \"e6b6e0be-d775-450d-b781-27d153aa722c\") "
Sep 15 19:23:04.203301 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203214    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6b6e0be-d775-450d-b781-27d153aa722c" (UID: "e6b6e0be-d775-450d-b781-27d153aa722c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:04.203343 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203288    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6b6e0be-d775-450d-b781-27d153aa722c" (UID: "e6b6e0be-d775-450d-b781-27d153aa722c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:04.203343 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203297    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-home" (OuterVolumeSpecName: "home") pod "e6b6e0be-d775-450d-b781-27d153aa722c" (UID: "e6b6e0be-d775-450d-b781-27d153aa722c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:04.203481 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203456    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:04.203481 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203484    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:04.203686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.203498    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:04.204975 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.204951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-dshm" (OuterVolumeSpecName: "dshm") pod "e6b6e0be-d775-450d-b781-27d153aa722c" (UID: "e6b6e0be-d775-450d-b781-27d153aa722c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:04.205055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.205031    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b6e0be-d775-450d-b781-27d153aa722c-kube-api-access-lfjd4" (OuterVolumeSpecName: "kube-api-access-lfjd4") pod "e6b6e0be-d775-450d-b781-27d153aa722c" (UID: "e6b6e0be-d775-450d-b781-27d153aa722c"). InnerVolumeSpecName "kube-api-access-lfjd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:23:04.205101 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.205080    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b6e0be-d775-450d-b781-27d153aa722c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6b6e0be-d775-450d-b781-27d153aa722c" (UID: "e6b6e0be-d775-450d-b781-27d153aa722c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:23:04.304020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.303997    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b6e0be-d775-450d-b781-27d153aa722c-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:04.304020 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.304019    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b6e0be-d775-450d-b781-27d153aa722c-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:04.304178 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.304029    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjd4\" (UniqueName: \"kubernetes.io/projected/e6b6e0be-d775-450d-b781-27d153aa722c-kube-api-access-lfjd4\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:04.816861 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.816822    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx" podUID="9d436bbe-0cd8-4149-8870-3cecfb0aff24" containerName="main" containerID="cri-o://d478c62e2f2c98fd1063d64d61cf11705f9ece840d7677b15201d752bc4ecf76" gracePeriod=45
Sep 15 19:23:04.941491 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.941452    2584 generic.go:356] "Generic (PLEG): container finished" podID="9d436bbe-0cd8-4149-8870-3cecfb0aff24" containerID="d478c62e2f2c98fd1063d64d61cf11705f9ece840d7677b15201d752bc4ecf76" exitCode=0
Sep 15 19:23:04.941943 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.941531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx" event={"ID":"9d436bbe-0cd8-4149-8870-3cecfb0aff24","Type":"ContainerDied","Data":"d478c62e2f2c98fd1063d64d61cf11705f9ece840d7677b15201d752bc4ecf76"}
Sep 15 19:23:04.943240 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.943214    2584 generic.go:356] "Generic (PLEG): container finished" podID="e6b6e0be-d775-450d-b781-27d153aa722c" containerID="91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968" exitCode=0
Sep 15 19:23:04.943387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.943262    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"
Sep 15 19:23:04.943387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.943305    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz" event={"ID":"e6b6e0be-d775-450d-b781-27d153aa722c","Type":"ContainerDied","Data":"91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968"}
Sep 15 19:23:04.943387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.943339    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz" event={"ID":"e6b6e0be-d775-450d-b781-27d153aa722c","Type":"ContainerDied","Data":"d201b8ecf79aeeb09a3bd957f1752af31e686c42cdc19fbf08a1750127c746e9"}
Sep 15 19:23:04.943387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.943359    2584 scope.go:117] "RemoveContainer" containerID="91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968"
Sep 15 19:23:04.958369 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.958349    2584 scope.go:117] "RemoveContainer" containerID="91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968"
Sep 15 19:23:04.958750 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:23:04.958724    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968\": container with ID starting with 91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968 not found: ID does not exist" containerID="91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968"
Sep 15 19:23:04.958863 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.958763    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968"} err="failed to get container status \"91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968\": rpc error: code = NotFound desc = could not find container \"91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968\": container with ID starting with 91fc5e98a03509323f1cec1f0187576b6ce0a2706f90ed066868a7c83d5ed968 not found: ID does not exist"
Sep 15 19:23:04.962106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.962079    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"]
Sep 15 19:23:04.968699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:04.968674    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c6d10a/sub-test-model-c6d10a-kserve-68f76c4f9c-ggfvz"]
Sep 15 19:23:05.081090 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.081034    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:23:05.214227 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214197    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tmp-dir\") pod \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") "
Sep 15 19:23:05.214417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214236    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tls-certs\") pod \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") "
Sep 15 19:23:05.214417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214289    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-home\") pod \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") "
Sep 15 19:23:05.214417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214321    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj7fm\" (UniqueName: \"kubernetes.io/projected/9d436bbe-0cd8-4149-8870-3cecfb0aff24-kube-api-access-vj7fm\") pod \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") "
Sep 15 19:23:05.214417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214337    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-model-cache\") pod \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") "
Sep 15 19:23:05.214417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214391    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-dshm\") pod \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\" (UID: \"9d436bbe-0cd8-4149-8870-3cecfb0aff24\") "
Sep 15 19:23:05.214724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214542    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d436bbe-0cd8-4149-8870-3cecfb0aff24" (UID: "9d436bbe-0cd8-4149-8870-3cecfb0aff24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:05.214724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214584    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-home" (OuterVolumeSpecName: "home") pod "9d436bbe-0cd8-4149-8870-3cecfb0aff24" (UID: "9d436bbe-0cd8-4149-8870-3cecfb0aff24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:05.214724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214622    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d436bbe-0cd8-4149-8870-3cecfb0aff24" (UID: "9d436bbe-0cd8-4149-8870-3cecfb0aff24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:05.214892 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214760    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:05.214892 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214782    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:05.214892 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.214794    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:05.216548 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.216515    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-dshm" (OuterVolumeSpecName: "dshm") pod "9d436bbe-0cd8-4149-8870-3cecfb0aff24" (UID: "9d436bbe-0cd8-4149-8870-3cecfb0aff24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:23:05.216685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.216592    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d436bbe-0cd8-4149-8870-3cecfb0aff24" (UID: "9d436bbe-0cd8-4149-8870-3cecfb0aff24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:23:05.216685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.216644    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d436bbe-0cd8-4149-8870-3cecfb0aff24-kube-api-access-vj7fm" (OuterVolumeSpecName: "kube-api-access-vj7fm") pod "9d436bbe-0cd8-4149-8870-3cecfb0aff24" (UID: "9d436bbe-0cd8-4149-8870-3cecfb0aff24"). InnerVolumeSpecName "kube-api-access-vj7fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:23:05.315387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.315360    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d436bbe-0cd8-4149-8870-3cecfb0aff24-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:05.315387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.315387    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7fm\" (UniqueName: \"kubernetes.io/projected/9d436bbe-0cd8-4149-8870-3cecfb0aff24-kube-api-access-vj7fm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:05.315546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.315400    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d436bbe-0cd8-4149-8870-3cecfb0aff24-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:23:05.952214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.952177    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx" event={"ID":"9d436bbe-0cd8-4149-8870-3cecfb0aff24","Type":"ContainerDied","Data":"76b8b8e876c0a31a88ea91f58b9a79b8f596a86d05751e3d985ec58d3e9ea806"}
Sep 15 19:23:05.952214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.952205    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"
Sep 15 19:23:05.952764 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.952237    2584 scope.go:117] "RemoveContainer" containerID="d478c62e2f2c98fd1063d64d61cf11705f9ece840d7677b15201d752bc4ecf76"
Sep 15 19:23:05.975346 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.975319    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"]
Sep 15 19:23:05.977734 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:05.977714    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/sub-test-model-415f87-kserve-5bf9865885-4blkx"]
Sep 15 19:23:06.234126 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:06.234045    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d436bbe-0cd8-4149-8870-3cecfb0aff24" path="/var/lib/kubelet/pods/9d436bbe-0cd8-4149-8870-3cecfb0aff24/volumes"
Sep 15 19:23:06.234505 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:06.234490    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b6e0be-d775-450d-b781-27d153aa722c" path="/var/lib/kubelet/pods/e6b6e0be-d775-450d-b781-27d153aa722c/volumes"
Sep 15 19:23:11.283516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283485    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"]
Sep 15 19:23:11.283914 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283901    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d436bbe-0cd8-4149-8870-3cecfb0aff24" containerName="main"
Sep 15 19:23:11.283958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283916    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d436bbe-0cd8-4149-8870-3cecfb0aff24" containerName="main"
Sep 15 19:23:11.283958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283927    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b6e0be-d775-450d-b781-27d153aa722c" containerName="main"
Sep 15 19:23:11.283958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283933    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b6e0be-d775-450d-b781-27d153aa722c" containerName="main"
Sep 15 19:23:11.283958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283951    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy"
Sep 15 19:23:11.283958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.283957    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4f654e-81b8-4480-a34c-cdb51f2831d3" containerName="istio-proxy"
Sep 15 19:23:11.284157 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.284015    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d436bbe-0cd8-4149-8870-3cecfb0aff24" containerName="main"
Sep 15 19:23:11.284157 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.284028    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b6e0be-d775-450d-b781-27d153aa722c" containerName="main"
Sep 15 19:23:11.288295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.288275    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.292112 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.292088    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"default-dockercfg-j4qsp\""
Sep 15 19:23:11.292228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.292087    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"rate-test-model-415f87-kserve-self-signed-certs\""
Sep 15 19:23:11.292228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.292125    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"kube-root-ca.crt\""
Sep 15 19:23:11.292228 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.292123    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-415f87\"/\"openshift-service-ca.crt\""
Sep 15 19:23:11.297403 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.297383    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"]
Sep 15 19:23:11.312465 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.312443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:11.474218 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.474177    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-model-cache\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.474401 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.474236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-home\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.474401 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.474363    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-dshm\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.474545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.474527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-tmp-dir\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.474602 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.474568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7wp\" (UniqueName: \"kubernetes.io/projected/3be6908c-fdf4-447e-a663-4e2bd71eb391-kube-api-access-pv7wp\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.474701 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.474669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6908c-fdf4-447e-a663-4e2bd71eb391-tls-certs\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-tmp-dir\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7wp\" (UniqueName: \"kubernetes.io/projected/3be6908c-fdf4-447e-a663-4e2bd71eb391-kube-api-access-pv7wp\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6908c-fdf4-447e-a663-4e2bd71eb391-tls-certs\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-model-cache\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-home\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-dshm\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-tmp-dir\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.575918 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-model-cache\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.576003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.575959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-home\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.577787 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.577764    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-dshm\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.577933 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.577916    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6908c-fdf4-447e-a663-4e2bd71eb391-tls-certs\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.583537 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.583516    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7wp\" (UniqueName: \"kubernetes.io/projected/3be6908c-fdf4-447e-a663-4e2bd71eb391-kube-api-access-pv7wp\") pod \"rate-test-model-415f87-kserve-797b85df7f-w8vcv\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") " pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.599417 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.599397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:11.876138 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.876113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"]
Sep 15 19:23:11.877948 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:23:11.877914    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3be6908c_fdf4_447e_a663_4e2bd71eb391.slice/crio-22d40beffe0e51cfb685f557022564c6bb93bc8dfea323447655222fc3ca4d4c WatchSource:0}: Error finding container 22d40beffe0e51cfb685f557022564c6bb93bc8dfea323447655222fc3ca4d4c: Status 404 returned error can't find the container with id 22d40beffe0e51cfb685f557022564c6bb93bc8dfea323447655222fc3ca4d4c
Sep 15 19:23:11.992193 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.992158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv" event={"ID":"3be6908c-fdf4-447e-a663-4e2bd71eb391","Type":"ContainerStarted","Data":"6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8"}
Sep 15 19:23:11.992340 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:11.992206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv" event={"ID":"3be6908c-fdf4-447e-a663-4e2bd71eb391","Type":"ContainerStarted","Data":"22d40beffe0e51cfb685f557022564c6bb93bc8dfea323447655222fc3ca4d4c"}
Sep 15 19:23:12.008921 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:12.008872    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv" podStartSLOduration=1.008855642 podStartE2EDuration="1.008855642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:23:12.007732286 +0000 UTC m=+1944.162100775" watchObservedRunningTime="2026-09-15 19:23:12.008855642 +0000 UTC m=+1944.163224124"
Sep 15 19:23:21.119415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:21.119379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:21.599591 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:21.599558    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:21.599802 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:21.599633    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:21.611650 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:21.611607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:22.076692 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:22.076666    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:23:47.830518 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:47.830439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:48.831006 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:48.830977    2584 scope.go:117] "RemoveContainer" containerID="f051055f4c44f69d6d362d9615aad2494e6358a979360c21c910ba663bea91ee"
Sep 15 19:23:48.840634 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:48.840587    2584 scope.go:117] "RemoveContainer" containerID="4f287b9f7140301bfca2174b477fe8acc1075d2ae871251c2eec96909a9abaf6"
Sep 15 19:23:48.848914 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:48.848898    2584 scope.go:117] "RemoveContainer" containerID="5f9ec48625f2969b62d6b72409a9ea2ec9740f87857471e7433a3f2fd827eb34"
Sep 15 19:23:48.882346 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:48.881636    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"]
Sep 15 19:23:48.904696 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:48.904605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:49.315325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.315297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:49.584597 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.584514    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"]
Sep 15 19:23:49.590131 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.590107    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.592503 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.592484    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f61404-openshift-default-dockercfg-v8xb2\""
Sep 15 19:23:49.601410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.601383    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"]
Sep 15 19:23:49.611297 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.611264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:49.712193 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.712162    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:49.727681 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903f1160-3e30-4b38-a207-18c12bd6abbb-istiod-ca-cert\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.727823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727693    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-data\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.727823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-socket\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.727823 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727805    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-envoy\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.727994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-credential-socket\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.727994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-token\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.727994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.727951    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-certs\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.728095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.728016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzd9\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-kube-api-access-bxzd9\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.728095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.728052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-podinfo\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.828781 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-socket\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.828971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828788    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-envoy\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.828971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-credential-socket\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.828971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-token\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.828971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-certs\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.828971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzd9\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-kube-api-access-bxzd9\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.828999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-podinfo\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903f1160-3e30-4b38-a207-18c12bd6abbb-istiod-ca-cert\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-data\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829171    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-socket\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-credential-socket\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-certs\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-data\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.829814 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.829797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903f1160-3e30-4b38-a207-18c12bd6abbb-istiod-ca-cert\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.831908 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.831873    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-envoy\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.832212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.832060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-podinfo\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.837964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.837907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-token\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.838044 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.837986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzd9\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-kube-api-access-bxzd9\") pod \"e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") " pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:49.902574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:49.902552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:50.180095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:50.180062    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"]
Sep 15 19:23:50.180510 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:23:50.180485    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903f1160_3e30_4b38_a207_18c12bd6abbb.slice/crio-93975965e3e00af907470a69cf567b993d7afa18b1ffb50112a57bc90e13821a WatchSource:0}: Error finding container 93975965e3e00af907470a69cf567b993d7afa18b1ffb50112a57bc90e13821a: Status 404 returned error can't find the container with id 93975965e3e00af907470a69cf567b993d7afa18b1ffb50112a57bc90e13821a
Sep 15 19:23:50.182756 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:50.182721    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:23:50.182834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:50.182788    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:23:50.182834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:50.182814    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:23:50.250625 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:50.250585    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" event={"ID":"903f1160-3e30-4b38-a207-18c12bd6abbb","Type":"ContainerStarted","Data":"93975965e3e00af907470a69cf567b993d7afa18b1ffb50112a57bc90e13821a"}
Sep 15 19:23:51.258363 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:51.258331    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" event={"ID":"903f1160-3e30-4b38-a207-18c12bd6abbb","Type":"ContainerStarted","Data":"6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"}
Sep 15 19:23:51.278164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:51.278110    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podStartSLOduration=2.278091958 podStartE2EDuration="2.278091958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:23:51.277552914 +0000 UTC m=+1983.431921391" watchObservedRunningTime="2026-09-15 19:23:51.278091958 +0000 UTC m=+1983.432460427"
Sep 15 19:23:51.903149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:51.903120    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:51.904361 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:51.904339    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Sep 15 19:23:51.904463 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:51.904385    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 15 19:23:52.521861 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:52.521825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:52.907065 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:52.906990    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:53.134542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:53.134510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:53.270746 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:53.270713    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:53.271711 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:53.271691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:23:53.328945 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:53.328913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:53.411905 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:53.411871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:54.052118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:54.052082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:54.755364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:54.755336    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:54.858466 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:54.858437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:54.957676 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:54.957646    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:55.261344 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:55.261315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:23:55.360551 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:23:55.360520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:03.907515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:03.907466    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv" podUID="3be6908c-fdf4-447e-a663-4e2bd71eb391" containerName="main" containerID="cri-o://6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8" gracePeriod=45
Sep 15 19:24:04.192840 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.192817    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:24:04.290520 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290493    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-home\") pod \"3be6908c-fdf4-447e-a663-4e2bd71eb391\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") "
Sep 15 19:24:04.290705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290567    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv7wp\" (UniqueName: \"kubernetes.io/projected/3be6908c-fdf4-447e-a663-4e2bd71eb391-kube-api-access-pv7wp\") pod \"3be6908c-fdf4-447e-a663-4e2bd71eb391\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") "
Sep 15 19:24:04.290705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290644    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-model-cache\") pod \"3be6908c-fdf4-447e-a663-4e2bd71eb391\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") "
Sep 15 19:24:04.290705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290670    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-tmp-dir\") pod \"3be6908c-fdf4-447e-a663-4e2bd71eb391\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") "
Sep 15 19:24:04.290705 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290702    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6908c-fdf4-447e-a663-4e2bd71eb391-tls-certs\") pod \"3be6908c-fdf4-447e-a663-4e2bd71eb391\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") "
Sep 15 19:24:04.290942 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290724    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-dshm\") pod \"3be6908c-fdf4-447e-a663-4e2bd71eb391\" (UID: \"3be6908c-fdf4-447e-a663-4e2bd71eb391\") "
Sep 15 19:24:04.290942 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290801    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-home" (OuterVolumeSpecName: "home") pod "3be6908c-fdf4-447e-a663-4e2bd71eb391" (UID: "3be6908c-fdf4-447e-a663-4e2bd71eb391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:04.290942 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290928    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-model-cache" (OuterVolumeSpecName: "model-cache") pod "3be6908c-fdf4-447e-a663-4e2bd71eb391" (UID: "3be6908c-fdf4-447e-a663-4e2bd71eb391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:04.291102 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.290942    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3be6908c-fdf4-447e-a663-4e2bd71eb391" (UID: "3be6908c-fdf4-447e-a663-4e2bd71eb391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:04.291164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.291119    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:24:04.291164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.291139    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:24:04.291164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.291155    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:24:04.292898 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.292874    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6908c-fdf4-447e-a663-4e2bd71eb391-kube-api-access-pv7wp" (OuterVolumeSpecName: "kube-api-access-pv7wp") pod "3be6908c-fdf4-447e-a663-4e2bd71eb391" (UID: "3be6908c-fdf4-447e-a663-4e2bd71eb391"). InnerVolumeSpecName "kube-api-access-pv7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:24:04.293130 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.293111    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be6908c-fdf4-447e-a663-4e2bd71eb391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3be6908c-fdf4-447e-a663-4e2bd71eb391" (UID: "3be6908c-fdf4-447e-a663-4e2bd71eb391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:24:04.293310 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.293265    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-dshm" (OuterVolumeSpecName: "dshm") pod "3be6908c-fdf4-447e-a663-4e2bd71eb391" (UID: "3be6908c-fdf4-447e-a663-4e2bd71eb391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:24:04.364814 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.364780    2584 generic.go:356] "Generic (PLEG): container finished" podID="3be6908c-fdf4-447e-a663-4e2bd71eb391" containerID="6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8" exitCode=0
Sep 15 19:24:04.364985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.364823    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"
Sep 15 19:24:04.364985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.364858    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv" event={"ID":"3be6908c-fdf4-447e-a663-4e2bd71eb391","Type":"ContainerDied","Data":"6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8"}
Sep 15 19:24:04.364985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.364904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv" event={"ID":"3be6908c-fdf4-447e-a663-4e2bd71eb391","Type":"ContainerDied","Data":"22d40beffe0e51cfb685f557022564c6bb93bc8dfea323447655222fc3ca4d4c"}
Sep 15 19:24:04.364985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.364924    2584 scope.go:117] "RemoveContainer" containerID="6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8"
Sep 15 19:24:04.374678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.374660    2584 scope.go:117] "RemoveContainer" containerID="6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8"
Sep 15 19:24:04.374944 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:24:04.374926    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8\": container with ID starting with 6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8 not found: ID does not exist" containerID="6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8"
Sep 15 19:24:04.374991 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.374952    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8"} err="failed to get container status \"6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8\": rpc error: code = NotFound desc = could not find container \"6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8\": container with ID starting with 6261be27e2e5e1c5a25f80659b6d1e0901bcd5debf5683075de10bc4390a7df8 not found: ID does not exist"
Sep 15 19:24:04.386350 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.386329    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"]
Sep 15 19:24:04.389781 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.389759    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-415f87/rate-test-model-415f87-kserve-797b85df7f-w8vcv"]
Sep 15 19:24:04.391904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.391887    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be6908c-fdf4-447e-a663-4e2bd71eb391-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:24:04.391904 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.391905    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be6908c-fdf4-447e-a663-4e2bd71eb391-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:24:04.392018 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:04.391913    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv7wp\" (UniqueName: \"kubernetes.io/projected/3be6908c-fdf4-447e-a663-4e2bd71eb391-kube-api-access-pv7wp\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:24:06.235196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:06.235159    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be6908c-fdf4-447e-a663-4e2bd71eb391" path="/var/lib/kubelet/pods/3be6908c-fdf4-447e-a663-4e2bd71eb391/volumes"
Sep 15 19:24:08.708912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:08.708675    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:52084->10.133.0.104:15021: read: connection reset by peer" start-of-body=
Sep 15 19:24:08.708912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:08.708763    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:52084->10.133.0.104:15021: read: connection reset by peer"
Sep 15 19:24:09.399566 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:09.399531    2584 generic.go:356] "Generic (PLEG): container finished" podID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerID="6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a" exitCode=0
Sep 15 19:24:09.399753 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:09.399603    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" event={"ID":"903f1160-3e30-4b38-a207-18c12bd6abbb","Type":"ContainerDied","Data":"6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"}
Sep 15 19:24:09.400025 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:09.400011    2584 scope.go:117] "RemoveContainer" containerID="6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"
Sep 15 19:24:09.401124 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:09.401091    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:09.401197 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:09.401170    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:09.401246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:09.401202    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:10.408270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:10.408235    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" event={"ID":"903f1160-3e30-4b38-a207-18c12bd6abbb","Type":"ContainerStarted","Data":"73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe"}
Sep 15 19:24:10.903654 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:10.903587    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:24:10.903961 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:10.903935    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Sep 15 19:24:10.904056 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:10.903992    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 15 19:24:11.903667 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:11.903606    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Sep 15 19:24:11.904062 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:11.903690    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 15 19:24:12.903127 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:12.903093    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Sep 15 19:24:12.903304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:12.903151    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 15 19:24:13.903265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:13.903219    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Sep 15 19:24:13.903679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:13.903291    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 15 19:24:14.903149 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:14.903110    2584 patch_prober.go:28] interesting pod/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Sep 15 19:24:14.903339 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:14.903172    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Sep 15 19:24:15.907035 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:15.907003    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:24:15.907483 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:15.907275    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:24:15.908057 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:15.908040    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:24:16.689549 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:16.689512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:17.216925 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:17.216889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:17.973499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:17.973461    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:18.073501 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:18.073470    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:18.631606 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:18.631570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:18.731954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:18.731916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:19.007083 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.007052    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"]
Sep 15 19:24:19.007717 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.007701    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be6908c-fdf4-447e-a663-4e2bd71eb391" containerName="main"
Sep 15 19:24:19.007791 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.007720    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be6908c-fdf4-447e-a663-4e2bd71eb391" containerName="main"
Sep 15 19:24:19.007844 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.007821    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be6908c-fdf4-447e-a663-4e2bd71eb391" containerName="main"
Sep 15 19:24:19.011419 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.011399    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.013857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.013836    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ce162-openshift-default-dockercfg-ghmls\""
Sep 15 19:24:19.025913 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.025885    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"]
Sep 15 19:24:19.047413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.047377    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:19.124337 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.124310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:19.142248 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-certs\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142404 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142287    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-socket\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142404 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzsf\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-kube-api-access-fmzsf\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142404 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026d25-042b-4529-92c8-4b345a5ae72e-istiod-ca-cert\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142461    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-data\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142511 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-credential-socket\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142546    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-istio-token\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142577 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142573    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-envoy\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.142671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.142595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026d25-042b-4529-92c8-4b345a5ae72e-istio-podinfo\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.243990 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.243960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026d25-042b-4529-92c8-4b345a5ae72e-istiod-ca-cert\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-data\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-credential-socket\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-istio-token\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-envoy\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026d25-042b-4529-92c8-4b345a5ae72e-istio-podinfo\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-certs\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244284    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-socket\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzsf\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-kube-api-access-fmzsf\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244499    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-data\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244821 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026d25-042b-4529-92c8-4b345a5ae72e-istiod-ca-cert\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-certs\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244856    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-socket\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.244915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.244875    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-credential-socket\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.246621 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.246580    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-envoy\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.246741 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.246719    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026d25-042b-4529-92c8-4b345a5ae72e-istio-podinfo\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.251682 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.251662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-istio-token\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.251682 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.251673    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzsf\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-kube-api-access-fmzsf\") pod \"e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") " pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.325628 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.325537    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:19.603223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.603194    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"]
Sep 15 19:24:19.604236 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:24:19.604207    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99026d25_042b_4529_92c8_4b345a5ae72e.slice/crio-4600809a68d67e0b92fd33247103606866d12ebd87a6b9c31a1f567bdd54fef1 WatchSource:0}: Error finding container 4600809a68d67e0b92fd33247103606866d12ebd87a6b9c31a1f567bdd54fef1: Status 404 returned error can't find the container with id 4600809a68d67e0b92fd33247103606866d12ebd87a6b9c31a1f567bdd54fef1
Sep 15 19:24:19.606798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.606762    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:19.606911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.606838    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:19.606911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:19.606883    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:20.481423 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:20.481386    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" event={"ID":"99026d25-042b-4529-92c8-4b345a5ae72e","Type":"ContainerStarted","Data":"862366baec3aaa42cf00535280b1991c78f75a3db8bc2a746b4dc2334941bc45"}
Sep 15 19:24:20.481892 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:20.481444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" event={"ID":"99026d25-042b-4529-92c8-4b345a5ae72e","Type":"ContainerStarted","Data":"4600809a68d67e0b92fd33247103606866d12ebd87a6b9c31a1f567bdd54fef1"}
Sep 15 19:24:20.501622 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:20.501556    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podStartSLOduration=2.501536714 podStartE2EDuration="2.501536714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:24:20.49984188 +0000 UTC m=+2012.654210351" watchObservedRunningTime="2026-09-15 19:24:20.501536714 +0000 UTC m=+2012.655905183"
Sep 15 19:24:21.326523 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:21.326482    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:21.331890 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:21.331865    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:21.487383 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:21.487351    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:21.488310 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:21.488288    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:22.226384 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:22.226323    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:22.366601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:22.366562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:22.456312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:22.456275    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:23.030295 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:23.030253    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:23.924187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:23.924149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:24.494042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:24.494009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:24.594489 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:24.594457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:25.067181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:25.067145    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:25.161732 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:25.161702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:25.342425 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:25.342345    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:40.611783 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:40.611700    2584 generic.go:356] "Generic (PLEG): container finished" podID="99026d25-042b-4529-92c8-4b345a5ae72e" containerID="862366baec3aaa42cf00535280b1991c78f75a3db8bc2a746b4dc2334941bc45" exitCode=0
Sep 15 19:24:40.612275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:40.611787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" event={"ID":"99026d25-042b-4529-92c8-4b345a5ae72e","Type":"ContainerDied","Data":"862366baec3aaa42cf00535280b1991c78f75a3db8bc2a746b4dc2334941bc45"}
Sep 15 19:24:40.612351 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:40.612272    2584 scope.go:117] "RemoveContainer" containerID="862366baec3aaa42cf00535280b1991c78f75a3db8bc2a746b4dc2334941bc45"
Sep 15 19:24:40.613360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:40.613331    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:40.613427 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:40.613413    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:40.613464 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:40.613446    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:24:41.619813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:41.619772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" event={"ID":"99026d25-042b-4529-92c8-4b345a5ae72e","Type":"ContainerStarted","Data":"fe4a0c9e6d65bda147a391413e2dbd32163568ce94178e1ba62562a9c6acd01c"}
Sep 15 19:24:42.326297 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:42.326254    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:42.326780 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:42.326745    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Sep 15 19:24:42.326911 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:42.326812    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Sep 15 19:24:43.326800 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:43.326753    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Sep 15 19:24:43.327304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:43.326823    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Sep 15 19:24:44.326501 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:44.326464    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Sep 15 19:24:44.326855 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:44.326821    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Sep 15 19:24:45.326533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:45.326494    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Sep 15 19:24:45.326830 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:45.326801    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Sep 15 19:24:45.963290 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:45.963255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:46.012757 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:46.012724    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:46.326468 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:46.326431    2584 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Sep 15 19:24:46.326790 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:46.326760    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Sep 15 19:24:46.415879 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:46.415847    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:46.947859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:46.947823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:47.298929 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:47.298896    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:47.427371 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:47.427332    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:47.427758 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:47.427728    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:47.428953 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:47.428928    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:24:47.723894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:47.723782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:48.429231 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:48.429200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:49.211715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:49.211607    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:53.768589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:53.768555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:53.875690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:53.875605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:24:58.652354 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:24:58.652312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:08.785497 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.785465    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"]
Sep 15 19:25:08.788062 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.788043    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:08.790813 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.790790    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f61404\"/\"ipp-route-f61404-kserve-self-signed-certs\""
Sep 15 19:25:08.792029 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.792007    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f61404\"/\"kube-root-ca.crt\""
Sep 15 19:25:08.792156 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.792099    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f61404\"/\"default-dockercfg-lsn87\""
Sep 15 19:25:08.792156 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.792006    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f61404\"/\"openshift-service-ca.crt\""
Sep 15 19:25:08.799123 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.799096    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"]
Sep 15 19:25:08.818478 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.818451    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:08.936095 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.936058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-tmp-dir\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:08.936443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.936419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139eff2-04f1-40e7-a172-f649293e90d6-tls-certs\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:08.936657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.936639    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-home\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:08.936837 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.936817    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwml\" (UniqueName: \"kubernetes.io/projected/8139eff2-04f1-40e7-a172-f649293e90d6-kube-api-access-cbwml\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:08.936979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.936964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-model-cache\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:08.937255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:08.937236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-dshm\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038198 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-dshm\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-tmp-dir\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139eff2-04f1-40e7-a172-f649293e90d6-tls-certs\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-home\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038368 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwml\" (UniqueName: \"kubernetes.io/projected/8139eff2-04f1-40e7-a172-f649293e90d6-kube-api-access-cbwml\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038594 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-model-cache\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038750 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038728    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-tmp-dir\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-home\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.038912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.038777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-model-cache\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.040797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.040773    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-dshm\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.040957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.040941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139eff2-04f1-40e7-a172-f649293e90d6-tls-certs\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.045960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.045935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwml\" (UniqueName: \"kubernetes.io/projected/8139eff2-04f1-40e7-a172-f649293e90d6-kube-api-access-cbwml\") pod \"ipp-route-f61404-kserve-87f7667bc-gvp7v\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") " pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.099939 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.099903    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:09.379688 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.379658    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"]
Sep 15 19:25:09.380590 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:25:09.380566    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8139eff2_04f1_40e7_a172_f649293e90d6.slice/crio-4698fdf08ffce2ad5b8868186db4108793701398da52bbbdb1e65ff02cd34fb1 WatchSource:0}: Error finding container 4698fdf08ffce2ad5b8868186db4108793701398da52bbbdb1e65ff02cd34fb1: Status 404 returned error can't find the container with id 4698fdf08ffce2ad5b8868186db4108793701398da52bbbdb1e65ff02cd34fb1
Sep 15 19:25:09.809771 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.809718    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v" event={"ID":"8139eff2-04f1-40e7-a172-f649293e90d6","Type":"ContainerStarted","Data":"d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e"}
Sep 15 19:25:09.810223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.809778    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v" event={"ID":"8139eff2-04f1-40e7-a172-f649293e90d6","Type":"ContainerStarted","Data":"4698fdf08ffce2ad5b8868186db4108793701398da52bbbdb1e65ff02cd34fb1"}
Sep 15 19:25:09.829604 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:09.829549    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v" podStartSLOduration=1.829531523 podStartE2EDuration="1.829531523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:25:09.827769228 +0000 UTC m=+2061.982137721" watchObservedRunningTime="2026-09-15 19:25:09.829531523 +0000 UTC m=+2061.983899991"
Sep 15 19:25:19.100895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:19.100863    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:19.101400 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:19.100913    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:19.113778 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:19.113753    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:19.885744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:19.885717    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:25:39.582202 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.582166    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:39.890774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.890702    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"]
Sep 15 19:25:39.893234 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.893208    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.895935 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.895912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f8b852-openshift-default-dockercfg-lknvb\""
Sep 15 19:25:39.912263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.912242    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"]
Sep 15 19:25:39.920752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.920727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:39.930662 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.930636    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-data\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.930881 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.930860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-credential-socket\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931043 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3102846b-a17e-4f53-b268-ddfed0e27c2a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931199 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-socket\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931314 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931298    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mdv\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-kube-api-access-95mdv\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931439 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-podinfo\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931571 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-envoy\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931767 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931749    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-token\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:39.931934 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:39.931902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-certs\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033086 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-podinfo\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-envoy\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-token\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-certs\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-data\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-credential-socket\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033292 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3102846b-a17e-4f53-b268-ddfed0e27c2a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033652 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-socket\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033652 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mdv\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-kube-api-access-95mdv\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033765 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-certs\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033897 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-data\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.033971 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-credential-socket\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.034029 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.033964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-socket\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.034238 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.034217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3102846b-a17e-4f53-b268-ddfed0e27c2a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.035475 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.035450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-envoy\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.035762 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.035744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-podinfo\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.042181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.042154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-token\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.042265 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.042230    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mdv\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-kube-api-access-95mdv\") pod \"e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") " pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.058066 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.058047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:40.204009 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.203929    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:40.486094 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.486069    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"]
Sep 15 19:25:40.487223 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:25:40.487195    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3102846b_a17e_4f53_b268_ddfed0e27c2a.slice/crio-33e2d755d482b31f301877703b66f60313471376af4bff6abf2eec5c49b96c23 WatchSource:0}: Error finding container 33e2d755d482b31f301877703b66f60313471376af4bff6abf2eec5c49b96c23: Status 404 returned error can't find the container with id 33e2d755d482b31f301877703b66f60313471376af4bff6abf2eec5c49b96c23
Sep 15 19:25:40.489447 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.489417    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:25:40.489519 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.489486    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:25:40.489555 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:40.489518    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:25:41.016459 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:41.016422    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" event={"ID":"3102846b-a17e-4f53-b268-ddfed0e27c2a","Type":"ContainerStarted","Data":"49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"}
Sep 15 19:25:41.016864 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:41.016483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" event={"ID":"3102846b-a17e-4f53-b268-ddfed0e27c2a","Type":"ContainerStarted","Data":"33e2d755d482b31f301877703b66f60313471376af4bff6abf2eec5c49b96c23"}
Sep 15 19:25:41.042701 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:41.042655    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podStartSLOduration=2.042636812 podStartE2EDuration="2.042636812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:25:41.040142062 +0000 UTC m=+2093.194510529" watchObservedRunningTime="2026-09-15 19:25:41.042636812 +0000 UTC m=+2093.197005280"
Sep 15 19:25:41.204677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:41.204647    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:41.210298 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:41.210273    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:42.024559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:42.024526    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:42.025461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:42.025431    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:25:42.248223 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:42.248193    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:43.920851 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:43.920820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:44.231396 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:44.231299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:44.968994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:44.968964    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:45.747735 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:45.747705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:46.490789 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:46.490757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:46.589318 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:46.589274    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:46.830886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:46.830853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:47.142492 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:47.142417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:25:47.266499 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:25:47.266465    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:01.149440 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:01.149405    2584 generic.go:356] "Generic (PLEG): container finished" podID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerID="49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b" exitCode=0
Sep 15 19:26:01.149880 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:01.149471    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" event={"ID":"3102846b-a17e-4f53-b268-ddfed0e27c2a","Type":"ContainerDied","Data":"49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"}
Sep 15 19:26:01.149925 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:01.149914    2584 scope.go:117] "RemoveContainer" containerID="49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"
Sep 15 19:26:01.151156 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:01.151127    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:26:01.151221 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:01.151199    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:26:01.151261 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:01.151232    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 19:26:02.158278 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:02.158234    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" event={"ID":"3102846b-a17e-4f53-b268-ddfed0e27c2a","Type":"ContainerStarted","Data":"636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b"}
Sep 15 19:26:02.204290 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:02.204256    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:26:02.204570 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:02.204549    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Sep 15 19:26:02.204647 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:02.204594    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Sep 15 19:26:03.204787 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:03.204751    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Sep 15 19:26:03.205236 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:03.204811    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Sep 15 19:26:04.204769 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:04.204731    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Sep 15 19:26:04.204974 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:04.204806    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Sep 15 19:26:05.204448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:05.204412    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Sep 15 19:26:05.204646 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:05.204475    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Sep 15 19:26:06.204448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:06.204414    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Sep 15 19:26:06.204860 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:06.204478    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Sep 15 19:26:07.204803 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:07.204768    2584 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Sep 15 19:26:07.205191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:07.204823    2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Sep 15 19:26:08.130859 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:08.130822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:08.208257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:08.208230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:26:08.208770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:08.208737    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:26:08.209463 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:08.209444    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:26:09.146699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:09.146670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:10.152975 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:10.152942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:10.856438 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:10.856400    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:11.625160 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:11.625126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:12.541231 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:12.541201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:13.125785 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.125755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:13.248464 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.248423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:13.648822 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.648788    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"]
Sep 15 19:26:13.651985 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.651969    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:13.654271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.654243    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:26:13.654380 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.654289    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:26:13.660816 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.660785    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"]
Sep 15 19:26:13.806150 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.806117    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc\" (UID: \"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:13.907478 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.907392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc\" (UID: \"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:13.908055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.908038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc\" (UID: \"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:13.964159 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:13.964131    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:14.046516 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:14.046428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:14.247668 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:14.247639    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"]
Sep 15 19:26:14.248477 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:26:14.248453    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de3a18a_cdcc_4681_94f6_1d34f2ed6dd1.slice/crio-3bc75a52679101f2c8b0a3ccfd2ba5ae05b8b45dbd502abb456fa1b7414d7c36 WatchSource:0}: Error finding container 3bc75a52679101f2c8b0a3ccfd2ba5ae05b8b45dbd502abb456fa1b7414d7c36: Status 404 returned error can't find the container with id 3bc75a52679101f2c8b0a3ccfd2ba5ae05b8b45dbd502abb456fa1b7414d7c36
Sep 15 19:26:15.069219 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:15.069184    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:15.255017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:15.254991    2584 generic.go:356] "Generic (PLEG): container finished" podID="7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1" containerID="f4d0d640e6fe9cac628455a407dfd85bb41f11640505c92a3b3896a29ac6ed7d" exitCode=0
Sep 15 19:26:15.255364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:15.255070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc" event={"ID":"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1","Type":"ContainerDied","Data":"f4d0d640e6fe9cac628455a407dfd85bb41f11640505c92a3b3896a29ac6ed7d"}
Sep 15 19:26:15.255364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:15.255119    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc" event={"ID":"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1","Type":"ContainerStarted","Data":"3bc75a52679101f2c8b0a3ccfd2ba5ae05b8b45dbd502abb456fa1b7414d7c36"}
Sep 15 19:26:15.744963 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:15.744877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:16.389533 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:16.389508    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:16.474781 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:16.474750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:16.535351 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:16.535239    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1-maas-api-service-ca\") pod \"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1\" (UID: \"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1\") "
Sep 15 19:26:16.536181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:16.536150    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1" (UID: "7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:16.637205 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:16.637169    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:17.269513 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:17.269486    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"
Sep 15 19:26:17.269728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:17.269486    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc" event={"ID":"7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1","Type":"ContainerDied","Data":"3bc75a52679101f2c8b0a3ccfd2ba5ae05b8b45dbd502abb456fa1b7414d7c36"}
Sep 15 19:26:17.269728 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:17.269622    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc75a52679101f2c8b0a3ccfd2ba5ae05b8b45dbd502abb456fa1b7414d7c36"
Sep 15 19:26:17.378885 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:17.378836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:18.123114 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:18.123074    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:24.318568 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:24.318534    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:24.418992 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:24.418964    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:24.520470 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:24.520433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:25.088203 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.088170    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:25.170454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.170423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:25.272375 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.272342    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:25.301875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.301843    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"]
Sep 15 19:26:25.306958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.306929    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f8b852-m2rfc"]
Sep 15 19:26:25.335686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.335658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:25.690849 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.690822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:25.793096 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:25.793062    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:26.234291 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:26.234253    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1" path="/var/lib/kubelet/pods/7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1/volumes"
Sep 15 19:26:26.440175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:26.440148    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:26.957775 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:26.957745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:27.742700 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:27.742662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:28.456557 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:28.456525    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:31.462528 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:31.462489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:31.577433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:31.577398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:31.629039 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:31.629002    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"]
Sep 15 19:26:31.629334 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:31.629303    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy" containerID="cri-o://636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b" gracePeriod=30
Sep 15 19:26:31.654153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:31.654126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:35.043063 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.043029    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"]
Sep 15 19:26:35.043474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.043460    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1" containerName="revoke-keys"
Sep 15 19:26:35.043474 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.043474    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1" containerName="revoke-keys"
Sep 15 19:26:35.043575 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.043561    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de3a18a-cdcc-4681-94f6-1d34f2ed6dd1" containerName="revoke-keys"
Sep 15 19:26:35.045954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.045937    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:35.048139 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.048120    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:26:35.048256 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.048177    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:26:35.054181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.054155    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"]
Sep 15 19:26:35.122586 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.122552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e753a12f-a53e-4b91-a21d-a9a29faffa5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx\" (UID: \"e753a12f-a53e-4b91-a21d-a9a29faffa5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:35.223523 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.223484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e753a12f-a53e-4b91-a21d-a9a29faffa5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx\" (UID: \"e753a12f-a53e-4b91-a21d-a9a29faffa5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:35.224170 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.224150    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e753a12f-a53e-4b91-a21d-a9a29faffa5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx\" (UID: \"e753a12f-a53e-4b91-a21d-a9a29faffa5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:35.324817 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.324712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:35.359779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.359746    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:35.638954 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.638916    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"]
Sep 15 19:26:35.640658 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:26:35.640598    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode753a12f_a53e_4b91_a21d_a9a29faffa5c.slice/crio-b417bc392adb14cbcfbe49b116c15f779af85bff25aa25bd2e71307e38b502b3 WatchSource:0}: Error finding container b417bc392adb14cbcfbe49b116c15f779af85bff25aa25bd2e71307e38b502b3: Status 404 returned error can't find the container with id b417bc392adb14cbcfbe49b116c15f779af85bff25aa25bd2e71307e38b502b3
Sep 15 19:26:35.961397 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:35.961368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:36.404550 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.404521    2584 generic.go:356] "Generic (PLEG): container finished" podID="e753a12f-a53e-4b91-a21d-a9a29faffa5c" containerID="96c1cd93f945dd0901cb9d1e16539d635a32572d1c3585d0bef1e59b4a5a3e9d" exitCode=0
Sep 15 19:26:36.404933 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.404591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx" event={"ID":"e753a12f-a53e-4b91-a21d-a9a29faffa5c","Type":"ContainerDied","Data":"96c1cd93f945dd0901cb9d1e16539d635a32572d1c3585d0bef1e59b4a5a3e9d"}
Sep 15 19:26:36.404933 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.404640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx" event={"ID":"e753a12f-a53e-4b91-a21d-a9a29faffa5c","Type":"ContainerStarted","Data":"b417bc392adb14cbcfbe49b116c15f779af85bff25aa25bd2e71307e38b502b3"}
Sep 15 19:26:36.641933 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.641884    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:36.889391 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.889360    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:26:36.941838 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941755    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-socket\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.941986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941863    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-token\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.941986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941922    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-podinfo\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.941986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-credential-socket\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.941986 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941969    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95mdv\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-kube-api-access-95mdv\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.942213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941995    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-envoy\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.942213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.941993    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:26:36.942213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-data\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.942213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942049    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3102846b-a17e-4f53-b268-ddfed0e27c2a-istiod-ca-cert\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.942213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942091    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-certs\") pod \"3102846b-a17e-4f53-b268-ddfed0e27c2a\" (UID: \"3102846b-a17e-4f53-b268-ddfed0e27c2a\") "
Sep 15 19:26:36.942469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942274    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:26:36.942589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942558    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:26:36.942666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942587    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:36.942666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942626    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:36.942800 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3102846b-a17e-4f53-b268-ddfed0e27c2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:36.942876 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.942846    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:26:36.944320 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.944269    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:26:36.944686 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.944652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:26:36.944818 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.944763    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-kube-api-access-95mdv" (OuterVolumeSpecName: "kube-api-access-95mdv") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "kube-api-access-95mdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:36.944885 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:36.944863    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3102846b-a17e-4f53-b268-ddfed0e27c2a" (UID: "3102846b-a17e-4f53-b268-ddfed0e27c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:26:37.043433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043400    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.043433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043428    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mdv\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-kube-api-access-95mdv\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.043656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043442    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.043656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043455    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.043656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043468    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3102846b-a17e-4f53-b268-ddfed0e27c2a-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.043656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043479    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3102846b-a17e-4f53-b268-ddfed0e27c2a-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.043656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.043489    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3102846b-a17e-4f53-b268-ddfed0e27c2a-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:37.300670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.300636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:37.414699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.414666    2584 generic.go:356] "Generic (PLEG): container finished" podID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerID="636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b" exitCode=0
Sep 15 19:26:37.415089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.414726    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"
Sep 15 19:26:37.415089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.414758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" event={"ID":"3102846b-a17e-4f53-b268-ddfed0e27c2a","Type":"ContainerDied","Data":"636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b"}
Sep 15 19:26:37.415089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.414817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z" event={"ID":"3102846b-a17e-4f53-b268-ddfed0e27c2a","Type":"ContainerDied","Data":"33e2d755d482b31f301877703b66f60313471376af4bff6abf2eec5c49b96c23"}
Sep 15 19:26:37.415089 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.414849    2584 scope.go:117] "RemoveContainer" containerID="636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b"
Sep 15 19:26:37.433398 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.433187    2584 scope.go:117] "RemoveContainer" containerID="49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"
Sep 15 19:26:37.440936 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.440857    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"]
Sep 15 19:26:37.443805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.443757    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f8b852-openshift-default-6656849bd6-t455z"]
Sep 15 19:26:37.445542 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.445480    2584 scope.go:117] "RemoveContainer" containerID="636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b"
Sep 15 19:26:37.446117 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:26:37.446073    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b\": container with ID starting with 636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b not found: ID does not exist" containerID="636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b"
Sep 15 19:26:37.446117 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.446102    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b"} err="failed to get container status \"636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b\": rpc error: code = NotFound desc = could not find container \"636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b\": container with ID starting with 636aa6fed745264569e1c50db0a7e63d2504a0d5fe490983d55d95b8c44c778b not found: ID does not exist"
Sep 15 19:26:37.446117 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.446120    2584 scope.go:117] "RemoveContainer" containerID="49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"
Sep 15 19:26:37.446438 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:26:37.446419    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b\": container with ID starting with 49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b not found: ID does not exist" containerID="49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"
Sep 15 19:26:37.446490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.446444    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b"} err="failed to get container status \"49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b\": rpc error: code = NotFound desc = could not find container \"49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b\": container with ID starting with 49e03e3694cb6db3d0b68f09fcfea5f351139da0936100dfe058ec6063aac15b not found: ID does not exist"
Sep 15 19:26:37.545367 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.545335    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:37.647899 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.647825    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e753a12f-a53e-4b91-a21d-a9a29faffa5c-maas-api-service-ca\") pod \"e753a12f-a53e-4b91-a21d-a9a29faffa5c\" (UID: \"e753a12f-a53e-4b91-a21d-a9a29faffa5c\") "
Sep 15 19:26:37.648189 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.648165    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e753a12f-a53e-4b91-a21d-a9a29faffa5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e753a12f-a53e-4b91-a21d-a9a29faffa5c" (UID: "e753a12f-a53e-4b91-a21d-a9a29faffa5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:26:37.749490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:37.749449    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e753a12f-a53e-4b91-a21d-a9a29faffa5c-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:26:38.017004 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:38.016966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:38.236573 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:38.236539    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" path="/var/lib/kubelet/pods/3102846b-a17e-4f53-b268-ddfed0e27c2a/volumes"
Sep 15 19:26:38.422679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:38.422577    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"
Sep 15 19:26:38.422679 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:38.422629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx" event={"ID":"e753a12f-a53e-4b91-a21d-a9a29faffa5c","Type":"ContainerDied","Data":"b417bc392adb14cbcfbe49b116c15f779af85bff25aa25bd2e71307e38b502b3"}
Sep 15 19:26:38.423179 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:38.422687    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b417bc392adb14cbcfbe49b116c15f779af85bff25aa25bd2e71307e38b502b3"
Sep 15 19:26:38.692079 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:38.691996    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:40.269835 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:40.269805    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:45.228486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:45.228393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:48.927217 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:48.927173    2584 scope.go:117] "RemoveContainer" containerID="0f39037757b60f1717ad3881db53785c70cb6cc44f57442c23c1c08f1e258842"
Sep 15 19:26:48.938948 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:48.938909    2584 scope.go:117] "RemoveContainer" containerID="9bdf538ad89a028a5c3ed4321d10cc4b283b6e8f2308d08c1403c7d8557da805"
Sep 15 19:26:50.855241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:50.855207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:50.955854 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:50.955820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:50.986936 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:50.986900    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"]
Sep 15 19:26:50.989966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:50.989940    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f61404-mf4tx"]
Sep 15 19:26:51.015402 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:51.015368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:51.353928 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:51.353894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:51.484410 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:51.484377    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:52.139145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:52.139111    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:52.236928 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:52.236883    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e753a12f-a53e-4b91-a21d-a9a29faffa5c" path="/var/lib/kubelet/pods/e753a12f-a53e-4b91-a21d-a9a29faffa5c/volumes"
Sep 15 19:26:52.683225 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:52.683181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:53.434592 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:53.434553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:26:56.334677 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:56.334639    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"]
Sep 15 19:26:56.385744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:26:56.385715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:09.651858 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:09.651819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:11.353861 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.353825    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v" podUID="8139eff2-04f1-40e7-a172-f649293e90d6" containerName="main" containerID="cri-o://d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e" gracePeriod=45
Sep 15 19:27:11.611682 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.611600    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:27:11.631116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.631085    2584 generic.go:356] "Generic (PLEG): container finished" podID="8139eff2-04f1-40e7-a172-f649293e90d6" containerID="d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e" exitCode=0
Sep 15 19:27:11.631264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.631136    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"
Sep 15 19:27:11.631264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.631175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v" event={"ID":"8139eff2-04f1-40e7-a172-f649293e90d6","Type":"ContainerDied","Data":"d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e"}
Sep 15 19:27:11.631264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.631221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v" event={"ID":"8139eff2-04f1-40e7-a172-f649293e90d6","Type":"ContainerDied","Data":"4698fdf08ffce2ad5b8868186db4108793701398da52bbbdb1e65ff02cd34fb1"}
Sep 15 19:27:11.631264 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.631239    2584 scope.go:117] "RemoveContainer" containerID="d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e"
Sep 15 19:27:11.643574 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.643551    2584 scope.go:117] "RemoveContainer" containerID="d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e"
Sep 15 19:27:11.643961 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:27:11.643930    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e\": container with ID starting with d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e not found: ID does not exist" containerID="d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e"
Sep 15 19:27:11.644060 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.643973    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e"} err="failed to get container status \"d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e\": rpc error: code = NotFound desc = could not find container \"d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e\": container with ID starting with d066e2e71908817e501393a176d6d79ab60b2fb660c865577f31f940963e919e not found: ID does not exist"
Sep 15 19:27:11.693769 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.693744    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139eff2-04f1-40e7-a172-f649293e90d6-tls-certs\") pod \"8139eff2-04f1-40e7-a172-f649293e90d6\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") "
Sep 15 19:27:11.693894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.693786    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-dshm\") pod \"8139eff2-04f1-40e7-a172-f649293e90d6\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") "
Sep 15 19:27:11.693894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.693820    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-model-cache\") pod \"8139eff2-04f1-40e7-a172-f649293e90d6\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") "
Sep 15 19:27:11.693894 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.693869    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-home\") pod \"8139eff2-04f1-40e7-a172-f649293e90d6\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") "
Sep 15 19:27:11.694038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.693971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbwml\" (UniqueName: \"kubernetes.io/projected/8139eff2-04f1-40e7-a172-f649293e90d6-kube-api-access-cbwml\") pod \"8139eff2-04f1-40e7-a172-f649293e90d6\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") "
Sep 15 19:27:11.694100 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694037    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-tmp-dir\") pod \"8139eff2-04f1-40e7-a172-f649293e90d6\" (UID: \"8139eff2-04f1-40e7-a172-f649293e90d6\") "
Sep 15 19:27:11.694181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694128    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8139eff2-04f1-40e7-a172-f649293e90d6" (UID: "8139eff2-04f1-40e7-a172-f649293e90d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:11.694246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694203    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-home" (OuterVolumeSpecName: "home") pod "8139eff2-04f1-40e7-a172-f649293e90d6" (UID: "8139eff2-04f1-40e7-a172-f649293e90d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:11.694304 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694277    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8139eff2-04f1-40e7-a172-f649293e90d6" (UID: "8139eff2-04f1-40e7-a172-f649293e90d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:11.694476 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694450    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:11.694476 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694479    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:11.694695 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.694489    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:11.698008 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.697981    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-dshm" (OuterVolumeSpecName: "dshm") pod "8139eff2-04f1-40e7-a172-f649293e90d6" (UID: "8139eff2-04f1-40e7-a172-f649293e90d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:11.698175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.698152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8139eff2-04f1-40e7-a172-f649293e90d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8139eff2-04f1-40e7-a172-f649293e90d6" (UID: "8139eff2-04f1-40e7-a172-f649293e90d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:27:11.698244 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.698155    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8139eff2-04f1-40e7-a172-f649293e90d6-kube-api-access-cbwml" (OuterVolumeSpecName: "kube-api-access-cbwml") pod "8139eff2-04f1-40e7-a172-f649293e90d6" (UID: "8139eff2-04f1-40e7-a172-f649293e90d6"). InnerVolumeSpecName "kube-api-access-cbwml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:11.795917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.795885    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbwml\" (UniqueName: \"kubernetes.io/projected/8139eff2-04f1-40e7-a172-f649293e90d6-kube-api-access-cbwml\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:11.795917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.795915    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139eff2-04f1-40e7-a172-f649293e90d6-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:11.795917 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.795925    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139eff2-04f1-40e7-a172-f649293e90d6-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:11.956390 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.956357    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"]
Sep 15 19:27:11.959883 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:11.959855    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f61404/ipp-route-f61404-kserve-87f7667bc-gvp7v"]
Sep 15 19:27:12.234490 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:12.234407    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8139eff2-04f1-40e7-a172-f649293e90d6" path="/var/lib/kubelet/pods/8139eff2-04f1-40e7-a172-f649293e90d6/volumes"
Sep 15 19:27:32.852204 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:32.852174    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:33.433916 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:33.433872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:38.867305 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:38.867273    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:38.967694 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:38.967662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:39.135181 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.135107    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"]
Sep 15 19:27:39.135712 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.135673    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy" containerID="cri-o://73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe" gracePeriod=30
Sep 15 19:27:39.158183 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.158159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:39.497262 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497184    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"]
Sep 15 19:27:39.497636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497621    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy"
Sep 15 19:27:39.497690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497636    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy"
Sep 15 19:27:39.497690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497652    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy"
Sep 15 19:27:39.497690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497658    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy"
Sep 15 19:27:39.497690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497665    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e753a12f-a53e-4b91-a21d-a9a29faffa5c" containerName="revoke-keys"
Sep 15 19:27:39.497690 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497671    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e753a12f-a53e-4b91-a21d-a9a29faffa5c" containerName="revoke-keys"
Sep 15 19:27:39.497852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497694    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8139eff2-04f1-40e7-a172-f649293e90d6" containerName="main"
Sep 15 19:27:39.497852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497700    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8139eff2-04f1-40e7-a172-f649293e90d6" containerName="main"
Sep 15 19:27:39.497852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497758    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8139eff2-04f1-40e7-a172-f649293e90d6" containerName="main"
Sep 15 19:27:39.497852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497766    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy"
Sep 15 19:27:39.497852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497770    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3102846b-a17e-4f53-b268-ddfed0e27c2a" containerName="istio-proxy"
Sep 15 19:27:39.497852 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.497780    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e753a12f-a53e-4b91-a21d-a9a29faffa5c" containerName="revoke-keys"
Sep 15 19:27:39.499863 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.499846    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:39.502153 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.502131    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:27:39.502269 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.502131    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:27:39.507803 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.507785    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"]
Sep 15 19:27:39.666504 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.666469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd\" (UID: \"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:39.702193 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.702164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:39.767937 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.767894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd\" (UID: \"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:39.768576 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.768550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd\" (UID: \"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:39.810929 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:39.810906    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:40.086656 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:40.086628    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"]
Sep 15 19:27:40.088250 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:27:40.088224    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c8751b_5e83_4a7e_a1bb_f0d0a4d6129d.slice/crio-fb05a883a17bf266299129f3baac4b6c61cd92f8f87658420bd55c921e42609d WatchSource:0}: Error finding container fb05a883a17bf266299129f3baac4b6c61cd92f8f87658420bd55c921e42609d: Status 404 returned error can't find the container with id fb05a883a17bf266299129f3baac4b6c61cd92f8f87658420bd55c921e42609d
Sep 15 19:27:40.399689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:40.399659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:40.810636 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:40.810572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd" event={"ID":"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d","Type":"ContainerStarted","Data":"ff8744ac3c91a079ba831357bc679bbba3b077c9361281aee9cddbfece470af1"}
Sep 15 19:27:40.810796 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:40.810660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd" event={"ID":"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d","Type":"ContainerStarted","Data":"fb05a883a17bf266299129f3baac4b6c61cd92f8f87658420bd55c921e42609d"}
Sep 15 19:27:40.826866 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:40.826819    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd" podStartSLOduration=1.82680359 podStartE2EDuration="1.82680359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:27:40.824563666 +0000 UTC m=+2212.978932137" watchObservedRunningTime="2026-09-15 19:27:40.82680359 +0000 UTC m=+2212.981172057"
Sep 15 19:27:41.218432 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:41.218351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:41.818084 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:41.818048    2584 generic.go:356] "Generic (PLEG): container finished" podID="e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d" containerID="ff8744ac3c91a079ba831357bc679bbba3b077c9361281aee9cddbfece470af1" exitCode=0
Sep 15 19:27:41.818300 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:41.818101    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd" event={"ID":"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d","Type":"ContainerDied","Data":"ff8744ac3c91a079ba831357bc679bbba3b077c9361281aee9cddbfece470af1"}
Sep 15 19:27:41.867709 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:41.867680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:42.417597 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:42.417566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:42.962111 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:42.962088    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:43.097488 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:43.097396    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d-maas-api-service-ca\") pod \"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d\" (UID: \"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d\") "
Sep 15 19:27:43.097765 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:43.097743    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d" (UID: "e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:43.198964 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:43.198939    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:43.832303 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:43.832265    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"
Sep 15 19:27:43.832703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:43.832319    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd" event={"ID":"e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d","Type":"ContainerDied","Data":"fb05a883a17bf266299129f3baac4b6c61cd92f8f87658420bd55c921e42609d"}
Sep 15 19:27:43.832703 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:43.832368    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb05a883a17bf266299129f3baac4b6c61cd92f8f87658420bd55c921e42609d"
Sep 15 19:27:44.400526 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.400472    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:27:44.408210 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408181    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-data\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408223    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-podinfo\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408248    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-credential-socket\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408266    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxzd9\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-kube-api-access-bxzd9\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408288    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-envoy\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408310    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-token\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408357 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408353    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903f1160-3e30-4b38-a207-18c12bd6abbb-istiod-ca-cert\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408416    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-socket\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408473    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:44.408724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-certs\") pod \"903f1160-3e30-4b38-a207-18c12bd6abbb\" (UID: \"903f1160-3e30-4b38-a207-18c12bd6abbb\") "
Sep 15 19:27:44.408724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408542    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:44.408960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408909    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.408960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.408929    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.409325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.409123    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:44.409325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.409125    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903f1160-3e30-4b38-a207-18c12bd6abbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:27:44.409325 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.409290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:44.410553 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.410527    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:27:44.410702 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.410677    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:44.410979 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.410958    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-kube-api-access-bxzd9" (OuterVolumeSpecName: "kube-api-access-bxzd9") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "kube-api-access-bxzd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:27:44.411106 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.411086    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "903f1160-3e30-4b38-a207-18c12bd6abbb" (UID: "903f1160-3e30-4b38-a207-18c12bd6abbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:27:44.510214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510187    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.510214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510214    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxzd9\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-kube-api-access-bxzd9\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.510389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510224    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.510389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510233    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903f1160-3e30-4b38-a207-18c12bd6abbb-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.510389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510241    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903f1160-3e30-4b38-a207-18c12bd6abbb-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.510389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510249    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.510389 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.510257    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903f1160-3e30-4b38-a207-18c12bd6abbb-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:27:44.840946 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.840909    2584 generic.go:356] "Generic (PLEG): container finished" podID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerID="73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe" exitCode=0
Sep 15 19:27:44.841322 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.840956    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"
Sep 15 19:27:44.841322 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.840991    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" event={"ID":"903f1160-3e30-4b38-a207-18c12bd6abbb","Type":"ContainerDied","Data":"73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe"}
Sep 15 19:27:44.841322 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.841046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw" event={"ID":"903f1160-3e30-4b38-a207-18c12bd6abbb","Type":"ContainerDied","Data":"93975965e3e00af907470a69cf567b993d7afa18b1ffb50112a57bc90e13821a"}
Sep 15 19:27:44.841322 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.841068    2584 scope.go:117] "RemoveContainer" containerID="73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe"
Sep 15 19:27:44.853898 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.853876    2584 scope.go:117] "RemoveContainer" containerID="6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"
Sep 15 19:27:44.863235 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.863219    2584 scope.go:117] "RemoveContainer" containerID="73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe"
Sep 15 19:27:44.863534 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:27:44.863515    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe\": container with ID starting with 73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe not found: ID does not exist" containerID="73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe"
Sep 15 19:27:44.863603 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.863546    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe"} err="failed to get container status \"73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe\": rpc error: code = NotFound desc = could not find container \"73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe\": container with ID starting with 73a65be678ec98028ec1525150aee567506eaa3e489a67c72d337343e32442fe not found: ID does not exist"
Sep 15 19:27:44.863603 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.863574    2584 scope.go:117] "RemoveContainer" containerID="6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"
Sep 15 19:27:44.863855 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:27:44.863838    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a\": container with ID starting with 6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a not found: ID does not exist" containerID="6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"
Sep 15 19:27:44.863895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.863860    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a"} err="failed to get container status \"6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a\": rpc error: code = NotFound desc = could not find container \"6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a\": container with ID starting with 6f858309fd3e91795c365b3ecf738088c7edf0a5c13c9f7410ab404a4068de6a not found: ID does not exist"
Sep 15 19:27:44.865682 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.865657    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"]
Sep 15 19:27:44.869038 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:44.869017    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f61404-openshift-default-7b686dd556-dfzlw"]
Sep 15 19:27:46.233517 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:46.233479    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" path="/var/lib/kubelet/pods/903f1160-3e30-4b38-a207-18c12bd6abbb/volumes"
Sep 15 19:27:48.969875 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:48.969844    2584 scope.go:117] "RemoveContainer" containerID="a005796b88777c7a9820ead4c7b65e4abc2b942bb8113d28cae0b207febdd979"
Sep 15 19:27:48.978489 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:48.978469    2584 scope.go:117] "RemoveContainer" containerID="9c554703d12efd9c0b2eef3fc2d5260d98214f3428af1ad73277d176b6550b9f"
Sep 15 19:27:48.986426 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:48.986409    2584 scope.go:117] "RemoveContainer" containerID="66a7e2d9db4aaf3751c3576e2e2fedd92bb9b7b54c7913cc5c6c307a8f87e6fe"
Sep 15 19:27:49.936866 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:49.936836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:50.011053 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.011019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:50.066815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.066779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:50.138137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.138107    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"]
Sep 15 19:27:50.139388 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.139361    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:50.140760 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.140740    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ce162-wskqd"]
Sep 15 19:27:50.197141 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.197058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:50.233805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.233774    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d" path="/var/lib/kubelet/pods/e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d/volumes"
Sep 15 19:27:50.487200 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.487126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:50.554853 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:50.554823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:51.122012 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:51.121974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:51.923321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:51.923272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:56.494589 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:56.494555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:56.594961 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:56.594933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:56.745448 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:56.745361    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"]
Sep 15 19:27:56.745671 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:56.745642    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy" containerID="cri-o://fe4a0c9e6d65bda147a391413e2dbd32163568ce94178e1ba62562a9c6acd01c" gracePeriod=30
Sep 15 19:27:56.770501 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:56.770467    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:57.105119 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105082    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"]
Sep 15 19:27:57.105559 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105541    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d" containerName="revoke-keys"
Sep 15 19:27:57.105664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105562    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d" containerName="revoke-keys"
Sep 15 19:27:57.105664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105583    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy"
Sep 15 19:27:57.105664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105592    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy"
Sep 15 19:27:57.105664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105634    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy"
Sep 15 19:27:57.105664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105643    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy"
Sep 15 19:27:57.105919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105750    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy"
Sep 15 19:27:57.105919 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.105766    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8751b-5e83-4a7e-a1bb-f0d0a4d6129d" containerName="revoke-keys"
Sep 15 19:27:57.108908 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.108887    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:27:57.112312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.112294    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:27:57.112646 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.112600    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:27:57.119484 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.119460    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"]
Sep 15 19:27:57.223833 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.223794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f9f2b2-679c-4234-9828-7a30d71e0961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq\" (UID: \"58f9f2b2-679c-4234-9828-7a30d71e0961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:27:57.324895 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.324857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f9f2b2-679c-4234-9828-7a30d71e0961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq\" (UID: \"58f9f2b2-679c-4234-9828-7a30d71e0961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:27:57.325653 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.325630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f9f2b2-679c-4234-9828-7a30d71e0961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq\" (UID: \"58f9f2b2-679c-4234-9828-7a30d71e0961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:27:57.392831 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.392753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:57.419719 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.419691    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:27:57.709182 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.709092    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"]
Sep 15 19:27:57.716271 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:27:57.716242    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f9f2b2_679c_4234_9828_7a30d71e0961.slice/crio-32df565153f10480a19c2a91c3aa3fca5d9ef7002708a9b67853dcc672cae459 WatchSource:0}: Error finding container 32df565153f10480a19c2a91c3aa3fca5d9ef7002708a9b67853dcc672cae459: Status 404 returned error can't find the container with id 32df565153f10480a19c2a91c3aa3fca5d9ef7002708a9b67853dcc672cae459
Sep 15 19:27:57.921317 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:57.921283    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq" event={"ID":"58f9f2b2-679c-4234-9828-7a30d71e0961","Type":"ContainerStarted","Data":"32df565153f10480a19c2a91c3aa3fca5d9ef7002708a9b67853dcc672cae459"}
Sep 15 19:27:58.171468 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:58.171440    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:58.837137 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:58.837053    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:27:58.930243 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:58.930206    2584 generic.go:356] "Generic (PLEG): container finished" podID="58f9f2b2-679c-4234-9828-7a30d71e0961" containerID="aaf25cfd5cb3db65cccef8bea30f59039aa47856e3c4eaa792b487ae7d50b570" exitCode=0
Sep 15 19:27:58.930415 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:58.930293    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq" event={"ID":"58f9f2b2-679c-4234-9828-7a30d71e0961","Type":"ContainerDied","Data":"aaf25cfd5cb3db65cccef8bea30f59039aa47856e3c4eaa792b487ae7d50b570"}
Sep 15 19:27:59.488798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:27:59.488767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:00.074759 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.074737    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:28:00.153191 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.153164    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f9f2b2-679c-4234-9828-7a30d71e0961-maas-api-service-ca\") pod \"58f9f2b2-679c-4234-9828-7a30d71e0961\" (UID: \"58f9f2b2-679c-4234-9828-7a30d71e0961\") "
Sep 15 19:28:00.153539 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.153518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f9f2b2-679c-4234-9828-7a30d71e0961-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58f9f2b2-679c-4234-9828-7a30d71e0961" (UID: "58f9f2b2-679c-4234-9828-7a30d71e0961"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:00.254201 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.254171    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f9f2b2-679c-4234-9828-7a30d71e0961-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:00.945000 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.944965    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"
Sep 15 19:28:00.945152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.945012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq" event={"ID":"58f9f2b2-679c-4234-9828-7a30d71e0961","Type":"ContainerDied","Data":"32df565153f10480a19c2a91c3aa3fca5d9ef7002708a9b67853dcc672cae459"}
Sep 15 19:28:00.945152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:00.945058    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32df565153f10480a19c2a91c3aa3fca5d9ef7002708a9b67853dcc672cae459"
Sep 15 19:28:01.953957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:01.953911    2584 generic.go:356] "Generic (PLEG): container finished" podID="99026d25-042b-4529-92c8-4b345a5ae72e" containerID="fe4a0c9e6d65bda147a391413e2dbd32163568ce94178e1ba62562a9c6acd01c" exitCode=0
Sep 15 19:28:01.954312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:01.953979    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" event={"ID":"99026d25-042b-4529-92c8-4b345a5ae72e","Type":"ContainerDied","Data":"fe4a0c9e6d65bda147a391413e2dbd32163568ce94178e1ba62562a9c6acd01c"}
Sep 15 19:28:01.954312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:01.954056    2584 scope.go:117] "RemoveContainer" containerID="862366baec3aaa42cf00535280b1991c78f75a3db8bc2a746b4dc2334941bc45"
Sep 15 19:28:01.998529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:01.998506    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:28:02.074327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074240    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-data\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074303    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026d25-042b-4529-92c8-4b345a5ae72e-istio-podinfo\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074340    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026d25-042b-4529-92c8-4b345a5ae72e-istiod-ca-cert\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074375    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-socket\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074441    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmzsf\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-kube-api-access-fmzsf\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074545 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074516    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-data" (OuterVolumeSpecName: "istio-data") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:02.074742 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074479    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-istio-token\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074749    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:02.074792 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074779    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-credential-socket\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074821    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-certs\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.074888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074777    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99026d25-042b-4529-92c8-4b345a5ae72e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:02.074888 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.074866    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-envoy\") pod \"99026d25-042b-4529-92c8-4b345a5ae72e\" (UID: \"99026d25-042b-4529-92c8-4b345a5ae72e\") "
Sep 15 19:28:02.075055 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075023    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:02.075103 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075048    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:02.075343 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075319    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.075452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075347    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.075452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075361    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.075452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075374    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99026d25-042b-4529-92c8-4b345a5ae72e-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.075452 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.075387    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.076593 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.076570    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99026d25-042b-4529-92c8-4b345a5ae72e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:28:02.076817 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.076789    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:02.077081 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.077060    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-kube-api-access-fmzsf" (OuterVolumeSpecName: "kube-api-access-fmzsf") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "kube-api-access-fmzsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:02.077164 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.077087    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-istio-token" (OuterVolumeSpecName: "istio-token") pod "99026d25-042b-4529-92c8-4b345a5ae72e" (UID: "99026d25-042b-4529-92c8-4b345a5ae72e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:02.176174 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.176123    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99026d25-042b-4529-92c8-4b345a5ae72e-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.176174 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.176168    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzsf\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-kube-api-access-fmzsf\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.176174 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.176180    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99026d25-042b-4529-92c8-4b345a5ae72e-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.176174 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.176189    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99026d25-042b-4529-92c8-4b345a5ae72e-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:02.960940 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.960858    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"
Sep 15 19:28:02.960940 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.960895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6" event={"ID":"99026d25-042b-4529-92c8-4b345a5ae72e","Type":"ContainerDied","Data":"4600809a68d67e0b92fd33247103606866d12ebd87a6b9c31a1f567bdd54fef1"}
Sep 15 19:28:02.961400 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.960955    2584 scope.go:117] "RemoveContainer" containerID="fe4a0c9e6d65bda147a391413e2dbd32163568ce94178e1ba62562a9c6acd01c"
Sep 15 19:28:02.980715 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.980690    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"]
Sep 15 19:28:02.987014 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:02.986994    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce162-openshift-default-7f4bcf7b66-7fpw6"]
Sep 15 19:28:04.233413 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:04.233374    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" path="/var/lib/kubelet/pods/99026d25-042b-4529-92c8-4b345a5ae72e/volumes"
Sep 15 19:28:07.631765 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:07.631732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:07.731524 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:07.731487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:07.808838 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:07.808807    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"]
Sep 15 19:28:07.811786 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:07.811762    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:07.813177 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:07.813159    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c6d10a-9l9dq"]
Sep 15 19:28:07.863640 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:07.863592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:08.163324 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:08.163296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:08.233246 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:08.233217    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f9f2b2-679c-4234-9828-7a30d71e0961" path="/var/lib/kubelet/pods/58f9f2b2-679c-4234-9828-7a30d71e0961/volumes"
Sep 15 19:28:08.264297 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:08.264265    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:08.849748 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:08.849719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:13.958345 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:13.958317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:14.058433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.058383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:14.186214 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.186177    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"]
Sep 15 19:28:14.186423 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.186399    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy" containerID="cri-o://959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39" gracePeriod=30
Sep 15 19:28:14.201770 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.201745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:14.591241 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591207    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"]
Sep 15 19:28:14.591603 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591590    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy"
Sep 15 19:28:14.591680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591604    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy"
Sep 15 19:28:14.591680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591639    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f9f2b2-679c-4234-9828-7a30d71e0961" containerName="revoke-keys"
Sep 15 19:28:14.591680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591646    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f9f2b2-679c-4234-9828-7a30d71e0961" containerName="revoke-keys"
Sep 15 19:28:14.591779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591704    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy"
Sep 15 19:28:14.591779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591713    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="903f1160-3e30-4b38-a207-18c12bd6abbb" containerName="istio-proxy"
Sep 15 19:28:14.591779 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.591724    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f9f2b2-679c-4234-9828-7a30d71e0961" containerName="revoke-keys"
Sep 15 19:28:14.596017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.595998    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:14.598841 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.598821    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9cqh\""
Sep 15 19:28:14.598965 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.598859    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:28:14.603860 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.603830    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"]
Sep 15 19:28:14.696060 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.696020    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc\" (UID: \"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:14.797364 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.797330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc\" (UID: \"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:14.798023 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.797997    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc\" (UID: \"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:14.837689 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.837664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:14.908033 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:14.907941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:15.180841 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:15.180812    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"]
Sep 15 19:28:15.182237 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:28:15.182209    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3fd1b2_d0e8_459b_8ddd_b86b70ab330c.slice/crio-11edbb625e234be8f93fae3a3494a29a29195b5cea9c5572cf1feaf4b764ecd7 WatchSource:0}: Error finding container 11edbb625e234be8f93fae3a3494a29a29195b5cea9c5572cf1feaf4b764ecd7: Status 404 returned error can't find the container with id 11edbb625e234be8f93fae3a3494a29a29195b5cea9c5572cf1feaf4b764ecd7
Sep 15 19:28:15.624428 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:15.624398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:16.045486 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:16.045453    2584 generic.go:356] "Generic (PLEG): container finished" podID="5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c" containerID="37e4a492bed17199fa44e2d5389e3141110653c751a5b54e67e72950441c1cd0" exitCode=0
Sep 15 19:28:16.045666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:16.045502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc" event={"ID":"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c","Type":"ContainerDied","Data":"37e4a492bed17199fa44e2d5389e3141110653c751a5b54e67e72950441c1cd0"}
Sep 15 19:28:16.045666 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:16.045544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc" event={"ID":"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c","Type":"ContainerStarted","Data":"11edbb625e234be8f93fae3a3494a29a29195b5cea9c5572cf1feaf4b764ecd7"}
Sep 15 19:28:16.145651 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:16.145604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:17.190469 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:17.190444    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:17.322990 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:17.322907    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c-maas-api-service-ca\") pod \"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c\" (UID: \"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c\") "
Sep 15 19:28:17.323281 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:17.323254    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c" (UID: "5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:17.424221 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:17.424190    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c-maas-api-service-ca\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:18.060795 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:18.060760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc" event={"ID":"5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c","Type":"ContainerDied","Data":"11edbb625e234be8f93fae3a3494a29a29195b5cea9c5572cf1feaf4b764ecd7"}
Sep 15 19:28:18.060955 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:18.060809    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11edbb625e234be8f93fae3a3494a29a29195b5cea9c5572cf1feaf4b764ecd7"
Sep 15 19:28:18.060955 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:18.060777    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"
Sep 15 19:28:19.445196 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.445173    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:28:19.544724 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544698    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istiod-ca-cert\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.544900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544759    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-certs\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.544900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544786    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btsmb\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-kube-api-access-btsmb\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.544900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544808    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-data\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.544900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544832    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-token\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.544900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544849    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-podinfo\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.544900 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544871    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-envoy\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.545229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544914    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-credential-socket\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.545229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.544940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-socket\") pod \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\" (UID: \"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a\") "
Sep 15 19:28:19.545229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545148    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:19.545229 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545099    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:19.545451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545338    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:19.545451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545376    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.545451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545392    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.545451 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545417    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:19.545714 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.545455    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:19.547048 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.547025    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:19.547048 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.547030    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:28:19.547195 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.547073    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:19.547195 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.547154    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-kube-api-access-btsmb" (OuterVolumeSpecName: "kube-api-access-btsmb") pod "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" (UID: "f9ef0663-efc7-4f01-b0f2-3edb3ee8316a"). InnerVolumeSpecName "kube-api-access-btsmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:19.646034 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646005    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsmb\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-kube-api-access-btsmb\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.646034 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646030    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.646184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646039    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.646184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646048    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.646184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646056    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.646184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646065    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:19.646184 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:19.646073    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:20.079225 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.079190    2584 generic.go:356] "Generic (PLEG): container finished" podID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerID="959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39" exitCode=0
Sep 15 19:28:20.079395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.079247    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"
Sep 15 19:28:20.079395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.079275    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" event={"ID":"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a","Type":"ContainerDied","Data":"959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39"}
Sep 15 19:28:20.079395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.079365    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8" event={"ID":"f9ef0663-efc7-4f01-b0f2-3edb3ee8316a","Type":"ContainerDied","Data":"eb42705c3d023f0e56276f694fa1745d66b40fe96815d234f684c2eb3d02b258"}
Sep 15 19:28:20.079395 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.079393    2584 scope.go:117] "RemoveContainer" containerID="959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39"
Sep 15 19:28:20.089680 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.089663    2584 scope.go:117] "RemoveContainer" containerID="2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"
Sep 15 19:28:20.098296 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.098279    2584 scope.go:117] "RemoveContainer" containerID="959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39"
Sep 15 19:28:20.098564 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:28:20.098545    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39\": container with ID starting with 959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39 not found: ID does not exist" containerID="959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39"
Sep 15 19:28:20.098637 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.098572    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39"} err="failed to get container status \"959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39\": rpc error: code = NotFound desc = could not find container \"959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39\": container with ID starting with 959f3d8cdfbfdb7979ce1527ded6587e96cb654ab810d2058bbcaec89b737e39 not found: ID does not exist"
Sep 15 19:28:20.098637 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.098588    2584 scope.go:117] "RemoveContainer" containerID="2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"
Sep 15 19:28:20.098874 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:28:20.098859    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef\": container with ID starting with 2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef not found: ID does not exist" containerID="2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"
Sep 15 19:28:20.098927 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.098877    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef"} err="failed to get container status \"2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef\": rpc error: code = NotFound desc = could not find container \"2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef\": container with ID starting with 2a7686d6a21b12a286ff71e8825530e5a16cd27b6aad17bf8aeae980ffe260ef not found: ID does not exist"
Sep 15 19:28:20.104633 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.104600    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"]
Sep 15 19:28:20.108143 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.108122    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c6d10a-openshift-default-6cb96cd5c6-d2gl8"]
Sep 15 19:28:20.234114 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:20.234078    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" path="/var/lib/kubelet/pods/f9ef0663-efc7-4f01-b0f2-3edb3ee8316a/volumes"
Sep 15 19:28:25.135958 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.135927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:25.231546 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.231508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:25.292837 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.292807    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"]
Sep 15 19:28:25.296271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.296246    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-415f87-5crfc"]
Sep 15 19:28:25.330144 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.330117    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:25.712283 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.712226    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:25.801189 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:25.801157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:26.234074 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:26.234041    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c" path="/var/lib/kubelet/pods/5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c/volumes"
Sep 15 19:28:31.407271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:31.407240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:31.507335 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:31.507304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:31.657915 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:31.657837    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"]
Sep 15 19:28:31.658071 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:31.658048    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" containerID="cri-o://469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9" gracePeriod=30
Sep 15 19:28:31.681936 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:31.681900    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:32.448649 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:32.448580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:28:34.623657 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:34.623596    2584 patch_prober.go:28] interesting pod/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 19:28:34.624013 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:34.623697    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 19:28:36.925052 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:36.924994    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:28:37.006011 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.005983    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-credential-socket\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006018    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-token\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006050    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-socket\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-podinfo\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-data\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006158    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8feba5e-d19d-458a-83dd-c37683f36e75-istiod-ca-cert\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l62sr\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-kube-api-access-l62sr\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006233    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-certs\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006225    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:37.006515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006350    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:37.006515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006365    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-envoy\") pod \"d8feba5e-d19d-458a-83dd-c37683f36e75\" (UID: \"d8feba5e-d19d-458a-83dd-c37683f36e75\") "
Sep 15 19:28:37.006515 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006496    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:37.006812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006567    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:37.006812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8feba5e-d19d-458a-83dd-c37683f36e75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 19:28:37.006812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006768    2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-credential-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.006812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006788    2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-socket\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.006812 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006804    2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-data\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.007069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006818    2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8feba5e-d19d-458a-83dd-c37683f36e75-istiod-ca-cert\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.007069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.006830    2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-workload-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.008224 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.008197    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:37.008387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.008369    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 19:28:37.008755 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.008740    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-kube-api-access-l62sr" (OuterVolumeSpecName: "kube-api-access-l62sr") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "kube-api-access-l62sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:28:37.008820 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.008756    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8feba5e-d19d-458a-83dd-c37683f36e75" (UID: "d8feba5e-d19d-458a-83dd-c37683f36e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:28:37.108155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.108115    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-token\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.108155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.108151    2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-podinfo\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.108155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.108160    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l62sr\" (UniqueName: \"kubernetes.io/projected/d8feba5e-d19d-458a-83dd-c37683f36e75-kube-api-access-l62sr\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.108155 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.108170    2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8feba5e-d19d-458a-83dd-c37683f36e75-istio-envoy\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:28:37.187120 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.187027    2584 generic.go:356] "Generic (PLEG): container finished" podID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerID="469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9" exitCode=0
Sep 15 19:28:37.187120 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.187078    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"
Sep 15 19:28:37.187120 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.187103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" event={"ID":"d8feba5e-d19d-458a-83dd-c37683f36e75","Type":"ContainerDied","Data":"469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9"}
Sep 15 19:28:37.187352 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.187156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d" event={"ID":"d8feba5e-d19d-458a-83dd-c37683f36e75","Type":"ContainerDied","Data":"faf7930814493c0b77e2ec2a17bcd2ccd52475cd22bdaa8206108e78cb6bc9b5"}
Sep 15 19:28:37.187352 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.187178    2584 scope.go:117] "RemoveContainer" containerID="469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9"
Sep 15 19:28:37.196978 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.196964    2584 scope.go:117] "RemoveContainer" containerID="e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"
Sep 15 19:28:37.205834 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.205797    2584 scope.go:117] "RemoveContainer" containerID="469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9"
Sep 15 19:28:37.206091 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:28:37.206074    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9\": container with ID starting with 469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9 not found: ID does not exist" containerID="469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9"
Sep 15 19:28:37.206163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.206099    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9"} err="failed to get container status \"469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9\": rpc error: code = NotFound desc = could not find container \"469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9\": container with ID starting with 469c76c4795cf3815098fbda9c1caa649e2c6ac11383adc14687925f103bc3d9 not found: ID does not exist"
Sep 15 19:28:37.206163 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.206115    2584 scope.go:117] "RemoveContainer" containerID="e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"
Sep 15 19:28:37.206333 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:28:37.206317    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd\": container with ID starting with e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd not found: ID does not exist" containerID="e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"
Sep 15 19:28:37.206376 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.206336    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd"} err="failed to get container status \"e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd\": rpc error: code = NotFound desc = could not find container \"e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd\": container with ID starting with e019a9b5dbb41eb9741d0fa566657e91ee0c410c7a103f998baa21cbd3eea3cd not found: ID does not exist"
Sep 15 19:28:37.208314 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.208294    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"]
Sep 15 19:28:37.212890 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:37.212869    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-415f87-openshift-default-f4c4c8cdb-64p2d"]
Sep 15 19:28:38.232805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:28:38.232776    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" path="/var/lib/kubelet/pods/d8feba5e-d19d-458a-83dd-c37683f36e75/volumes"
Sep 15 19:29:14.286053 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:29:14.286011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:29:56.697912 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:29:56.697836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:29:58.312327 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:29:58.312293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:30:04.385420 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:30:04.385387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:30:04.485866 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:30:04.485824    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:30:16.100664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:30:16.100606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:31:10.819642 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:10.819562    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"]
Sep 15 19:31:10.820190 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:10.819857    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj" podUID="938cd1dd-3ac6-4af7-86e7-eea329eda610" containerName="manager" containerID="cri-o://7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a" gracePeriod=10
Sep 15 19:31:11.091658 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.091626    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:31:11.133580 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.133544    2584 generic.go:356] "Generic (PLEG): container finished" podID="938cd1dd-3ac6-4af7-86e7-eea329eda610" containerID="7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a" exitCode=0
Sep 15 19:31:11.133776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.133603    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"
Sep 15 19:31:11.133776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.133651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj" event={"ID":"938cd1dd-3ac6-4af7-86e7-eea329eda610","Type":"ContainerDied","Data":"7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a"}
Sep 15 19:31:11.133776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.133722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj" event={"ID":"938cd1dd-3ac6-4af7-86e7-eea329eda610","Type":"ContainerDied","Data":"ccd72ef7f7c58452ff3568ffd066fe27e01267b4b646e65a2630e08b208bc0cf"}
Sep 15 19:31:11.133776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.133755    2584 scope.go:117] "RemoveContainer" containerID="7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a"
Sep 15 19:31:11.146016 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.145990    2584 scope.go:117] "RemoveContainer" containerID="7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a"
Sep 15 19:31:11.146393 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:31:11.146358    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a\": container with ID starting with 7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a not found: ID does not exist" containerID="7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a"
Sep 15 19:31:11.146498 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.146409    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a"} err="failed to get container status \"7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a\": rpc error: code = NotFound desc = could not find container \"7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a\": container with ID starting with 7bea57b941131cefe6ac70206458e9426ab9f12deb7ed2661da6a8851658a96a not found: ID does not exist"
Sep 15 19:31:11.184926 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.184875    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjvbb\" (UniqueName: \"kubernetes.io/projected/938cd1dd-3ac6-4af7-86e7-eea329eda610-kube-api-access-wjvbb\") pod \"938cd1dd-3ac6-4af7-86e7-eea329eda610\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") "
Sep 15 19:31:11.185088 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.184989    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938cd1dd-3ac6-4af7-86e7-eea329eda610-extensions-socket-volume\") pod \"938cd1dd-3ac6-4af7-86e7-eea329eda610\" (UID: \"938cd1dd-3ac6-4af7-86e7-eea329eda610\") "
Sep 15 19:31:11.185378 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.185346    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938cd1dd-3ac6-4af7-86e7-eea329eda610-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "938cd1dd-3ac6-4af7-86e7-eea329eda610" (UID: "938cd1dd-3ac6-4af7-86e7-eea329eda610"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:31:11.187165 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.187137    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938cd1dd-3ac6-4af7-86e7-eea329eda610-kube-api-access-wjvbb" (OuterVolumeSpecName: "kube-api-access-wjvbb") pod "938cd1dd-3ac6-4af7-86e7-eea329eda610" (UID: "938cd1dd-3ac6-4af7-86e7-eea329eda610"). InnerVolumeSpecName "kube-api-access-wjvbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:31:11.285973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.285922    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvbb\" (UniqueName: \"kubernetes.io/projected/938cd1dd-3ac6-4af7-86e7-eea329eda610-kube-api-access-wjvbb\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:31:11.285973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.285961    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938cd1dd-3ac6-4af7-86e7-eea329eda610-extensions-socket-volume\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:31:11.456627 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.456561    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"]
Sep 15 19:31:11.460187 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:11.460148    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5jswj"]
Sep 15 19:31:12.234321 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:12.234273    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938cd1dd-3ac6-4af7-86e7-eea329eda610" path="/var/lib/kubelet/pods/938cd1dd-3ac6-4af7-86e7-eea329eda610/volumes"
Sep 15 19:31:27.899271 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:27.899232    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:31:32.080699 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:32.080661    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:31:32.183176 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:31:32.183142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:32:11.500803 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.500764    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"]
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501135    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938cd1dd-3ac6-4af7-86e7-eea329eda610" containerName="manager"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501148    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="938cd1dd-3ac6-4af7-86e7-eea329eda610" containerName="manager"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501157    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501164    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501178    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501183    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501189    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501194    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501200    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501205    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501216    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c" containerName="revoke-keys"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501221    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c" containerName="revoke-keys"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501232    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy"
Sep 15 19:32:11.501270 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501237    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy"
Sep 15 19:32:11.501740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501306    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy"
Sep 15 19:32:11.501740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501315    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3fd1b2-d0e8-459b-8ddd-b86b70ab330c" containerName="revoke-keys"
Sep 15 19:32:11.501740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501321    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy"
Sep 15 19:32:11.501740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501326    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="99026d25-042b-4529-92c8-4b345a5ae72e" containerName="istio-proxy"
Sep 15 19:32:11.501740 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.501331    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="938cd1dd-3ac6-4af7-86e7-eea329eda610" containerName="manager"
Sep 15 19:32:11.504473 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.504445    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.506800 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.506774    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tk6j7\""
Sep 15 19:32:11.515788 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.515757    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"]
Sep 15 19:32:11.558433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.558385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k695n\" (UniqueName: \"kubernetes.io/projected/61ad4629-6f4b-44f2-a96f-0bb7fc54d50a-kube-api-access-k695n\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lbffm\" (UID: \"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.558659 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.558491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61ad4629-6f4b-44f2-a96f-0bb7fc54d50a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lbffm\" (UID: \"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.659470 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.659404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k695n\" (UniqueName: \"kubernetes.io/projected/61ad4629-6f4b-44f2-a96f-0bb7fc54d50a-kube-api-access-k695n\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lbffm\" (UID: \"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.659707 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.659502    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61ad4629-6f4b-44f2-a96f-0bb7fc54d50a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lbffm\" (UID: \"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.659994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.659974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61ad4629-6f4b-44f2-a96f-0bb7fc54d50a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lbffm\" (UID: \"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.667362 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.667332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k695n\" (UniqueName: \"kubernetes.io/projected/61ad4629-6f4b-44f2-a96f-0bb7fc54d50a-kube-api-access-k695n\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lbffm\" (UID: \"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:11.815405 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:11.815360    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:12.100340 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:12.100305    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"]
Sep 15 19:32:12.102730 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:32:12.102702    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ad4629_6f4b_44f2_a96f_0bb7fc54d50a.slice/crio-d7d66be9c4829af7c07afda0ffa1ffed63946f11d76f70b5809eed4d5c3e8303 WatchSource:0}: Error finding container d7d66be9c4829af7c07afda0ffa1ffed63946f11d76f70b5809eed4d5c3e8303: Status 404 returned error can't find the container with id d7d66be9c4829af7c07afda0ffa1ffed63946f11d76f70b5809eed4d5c3e8303
Sep 15 19:32:12.521814 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:12.521769    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm" event={"ID":"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a","Type":"ContainerStarted","Data":"1a8e0a2b64f9e93f47270666c5fc976f6138e3b1b49de74c9c6eaf40e1a3be72"}
Sep 15 19:32:12.522312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:12.521854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm" event={"ID":"61ad4629-6f4b-44f2-a96f-0bb7fc54d50a","Type":"ContainerStarted","Data":"d7d66be9c4829af7c07afda0ffa1ffed63946f11d76f70b5809eed4d5c3e8303"}
Sep 15 19:32:12.522312 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:12.522067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:12.545594 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:12.545529    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm" podStartSLOduration=1.5455098569999999 podStartE2EDuration="1.545509857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:32:12.543471669 +0000 UTC m=+2484.697840134" watchObservedRunningTime="2026-09-15 19:32:12.545509857 +0000 UTC m=+2484.699878325"
Sep 15 19:32:15.089564 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:15.089527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:32:23.530352 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:23.530313    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lbffm"
Sep 15 19:32:49.080504 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:49.080420    2584 scope.go:117] "RemoveContainer" containerID="f4d0d640e6fe9cac628455a407dfd85bb41f11640505c92a3b3896a29ac6ed7d"
Sep 15 19:32:49.089965 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:49.089932    2584 scope.go:117] "RemoveContainer" containerID="96c1cd93f945dd0901cb9d1e16539d635a32572d1c3585d0bef1e59b4a5a3e9d"
Sep 15 19:32:54.279835 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.279798    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"]
Sep 15 19:32:54.280284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.280247    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ef0663-efc7-4f01-b0f2-3edb3ee8316a" containerName="istio-proxy"
Sep 15 19:32:54.280284 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.280266    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8feba5e-d19d-458a-83dd-c37683f36e75" containerName="istio-proxy"
Sep 15 19:32:54.284572 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.284545    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.287053 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.287019    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b88ee092-kserve-self-signed-certs\""
Sep 15 19:32:54.304194 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.304152    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"]
Sep 15 19:32:54.305865 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.305818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:32:54.372845 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.372802    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-dshm\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.373029 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.372873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-model-cache\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.373076 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.373058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tmp-dir\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.373118 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.373087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dkg\" (UniqueName: \"kubernetes.io/projected/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-kube-api-access-r7dkg\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.373255 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.373228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-home\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.373306 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.373284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tls-certs\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474384 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-model-cache\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474384 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tmp-dir\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dkg\" (UniqueName: \"kubernetes.io/projected/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-kube-api-access-r7dkg\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474457    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-home\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tls-certs\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474670 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-dshm\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474910 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474890    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-model-cache\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474975 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-home\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.474975 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.474945    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tmp-dir\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.476920 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.476894    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-dshm\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.477213 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.477189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tls-certs\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.482471 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.482442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dkg\" (UniqueName: \"kubernetes.io/projected/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-kube-api-access-r7dkg\") pod \"e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") " pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.598465 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.598375    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:32:54.873209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:54.873176    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"]
Sep 15 19:32:54.874507 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:32:54.874484    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod637fbf24_08a2_4d9b_88f7_c0d09c3ed7b9.slice/crio-6a61c6df02ab2d34a99a545ec52933bc8dcad990e1942ed1ddc92fa67049f2d8 WatchSource:0}: Error finding container 6a61c6df02ab2d34a99a545ec52933bc8dcad990e1942ed1ddc92fa67049f2d8: Status 404 returned error can't find the container with id 6a61c6df02ab2d34a99a545ec52933bc8dcad990e1942ed1ddc92fa67049f2d8
Sep 15 19:32:55.790431 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:55.790392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8" event={"ID":"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9","Type":"ContainerStarted","Data":"576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb"}
Sep 15 19:32:55.790957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:55.790446    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8" event={"ID":"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9","Type":"ContainerStarted","Data":"6a61c6df02ab2d34a99a545ec52933bc8dcad990e1942ed1ddc92fa67049f2d8"}
Sep 15 19:32:55.807684 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:32:55.807627    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8" podStartSLOduration=1.80759383 podStartE2EDuration="1.80759383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:32:55.806837817 +0000 UTC m=+2527.961206282" watchObservedRunningTime="2026-09-15 19:32:55.80759383 +0000 UTC m=+2527.961962298"
Sep 15 19:33:04.599227 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:04.599184    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:33:04.599805 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:04.599253    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:33:04.612218 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:04.612189    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:33:04.856427 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:04.856335    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:33:11.042798 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:11.042760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:33:42.302561 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:42.302517    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"]
Sep 15 19:33:42.322209 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:42.322169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:33:43.231764 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:43.231719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:33:49.122579 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:49.122543    2584 scope.go:117] "RemoveContainer" containerID="ff8744ac3c91a079ba831357bc679bbba3b077c9361281aee9cddbfece470af1"
Sep 15 19:33:57.323431 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.323378    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8" podUID="637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" containerName="main" containerID="cri-o://576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb" gracePeriod=45
Sep 15 19:33:57.583623 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.583535    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:33:57.687973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.687917    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-dshm\") pod \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") "
Sep 15 19:33:57.687973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.687978    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7dkg\" (UniqueName: \"kubernetes.io/projected/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-kube-api-access-r7dkg\") pod \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") "
Sep 15 19:33:57.688212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688039    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-model-cache\") pod \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") "
Sep 15 19:33:57.688212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688093    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tmp-dir\") pod \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") "
Sep 15 19:33:57.688212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688122    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-home\") pod \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") "
Sep 15 19:33:57.688212 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688156    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tls-certs\") pod \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\" (UID: \"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9\") "
Sep 15 19:33:57.688433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688348    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" (UID: "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:57.688433 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688411    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" (UID: "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:57.688539 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.688505    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-home" (OuterVolumeSpecName: "home") pod "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" (UID: "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:57.690439 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.690407    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" (UID: "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 19:33:57.690664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.690471    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-kube-api-access-r7dkg" (OuterVolumeSpecName: "kube-api-access-r7dkg") pod "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" (UID: "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9"). InnerVolumeSpecName "kube-api-access-r7dkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 19:33:57.690664 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.690497    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-dshm" (OuterVolumeSpecName: "dshm") pod "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" (UID: "637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 19:33:57.789500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.789444    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tmp-dir\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:33:57.789500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.789478    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-home\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:33:57.789500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.789487    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-tls-certs\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:33:57.789500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.789494    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-dshm\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:33:57.789500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.789502    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dkg\" (UniqueName: \"kubernetes.io/projected/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-kube-api-access-r7dkg\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:33:57.789500 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:57.789512    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9-model-cache\") on node \"ip-10-0-143-252.ec2.internal\" DevicePath \"\""
Sep 15 19:33:58.204394 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.204357    2584 generic.go:356] "Generic (PLEG): container finished" podID="637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" containerID="576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb" exitCode=0
Sep 15 19:33:58.204578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.204423    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"
Sep 15 19:33:58.204578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.204446    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8" event={"ID":"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9","Type":"ContainerDied","Data":"576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb"}
Sep 15 19:33:58.204578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.204499    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8" event={"ID":"637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9","Type":"ContainerDied","Data":"6a61c6df02ab2d34a99a545ec52933bc8dcad990e1942ed1ddc92fa67049f2d8"}
Sep 15 19:33:58.204578 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.204521    2584 scope.go:117] "RemoveContainer" containerID="576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb"
Sep 15 19:33:58.217489 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.217462    2584 scope.go:117] "RemoveContainer" containerID="576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb"
Sep 15 19:33:58.217892 ip-10-0-143-252 kubenswrapper[2584]: E0915 19:33:58.217866    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb\": container with ID starting with 576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb not found: ID does not exist" containerID="576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb"
Sep 15 19:33:58.217960 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.217903    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb"} err="failed to get container status \"576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb\": rpc error: code = NotFound desc = could not find container \"576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb\": container with ID starting with 576cbe9dcf5965897244c19a0196d80d19cd218635ec96e73161160672f392eb not found: ID does not exist"
Sep 15 19:33:58.227152 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.227121    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"]
Sep 15 19:33:58.234257 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:33:58.234225    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b88ee092-kserve-6c8ddfdd5f-nkfj8"]
Sep 15 19:34:00.234387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:00.234345    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" path="/var/lib/kubelet/pods/637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9/volumes"
Sep 15 19:34:08.191423 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.191387    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"]
Sep 15 19:34:08.191886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.191870    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" containerName="main"
Sep 15 19:34:08.191886 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.191888    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" containerName="main"
Sep 15 19:34:08.191966 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.191953    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="637fbf24-08a2-4d9b-88f7-c0d09c3ed7b9" containerName="main"
Sep 15 19:34:08.196466 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.196440    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.198846 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.198812    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 19:34:08.199737 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.199716    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 19:34:08.199869 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.199852    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 19:34:08.199975 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.199954    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v55qf\""
Sep 15 19:34:08.200104 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.200028    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 19:34:08.206847 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.206815    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"]
Sep 15 19:34:08.292206 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.292163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/915b8202-db44-4328-833d-8968478aff2e-maas-api-tls\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.292406 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.292229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/915b8202-db44-4328-833d-8968478aff2e-metrics-certs\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.292406 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.292324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/915b8202-db44-4328-833d-8968478aff2e-kube-api-access-r4xxj\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.292406 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.292376    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/915b8202-db44-4328-833d-8968478aff2e-openshift-service-ca\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.393774 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.393731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/915b8202-db44-4328-833d-8968478aff2e-metrics-certs\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.393972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.393897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/915b8202-db44-4328-833d-8968478aff2e-kube-api-access-r4xxj\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.393972 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.393951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/915b8202-db44-4328-833d-8968478aff2e-openshift-service-ca\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.394088 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.394008    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/915b8202-db44-4328-833d-8968478aff2e-maas-api-tls\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.394655 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.394594    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/915b8202-db44-4328-833d-8968478aff2e-openshift-service-ca\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.396651 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.396605    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/915b8202-db44-4328-833d-8968478aff2e-maas-api-tls\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.396776 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.396605    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/915b8202-db44-4328-833d-8968478aff2e-metrics-certs\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.402678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.402652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xxj\" (UniqueName: \"kubernetes.io/projected/915b8202-db44-4328-833d-8968478aff2e-kube-api-access-r4xxj\") pod \"maas-api-86d9ddc875-l292q\" (UID: \"915b8202-db44-4328-833d-8968478aff2e\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.509968 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.509919    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:08.787145 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.787110    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"]
Sep 15 19:34:08.788797 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:34:08.788766    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915b8202_db44_4328_833d_8968478aff2e.slice/crio-e3cc9dd4debed8f7b19001a61dd90a7902545f5a2c210509a1a45fd537647126 WatchSource:0}: Error finding container e3cc9dd4debed8f7b19001a61dd90a7902545f5a2c210509a1a45fd537647126: Status 404 returned error can't find the container with id e3cc9dd4debed8f7b19001a61dd90a7902545f5a2c210509a1a45fd537647126
Sep 15 19:34:08.790842 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:08.790816    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 19:34:09.283744 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:09.283698    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q" event={"ID":"915b8202-db44-4328-833d-8968478aff2e","Type":"ContainerStarted","Data":"e3cc9dd4debed8f7b19001a61dd90a7902545f5a2c210509a1a45fd537647126"}
Sep 15 19:34:11.304042 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:11.304004    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q" event={"ID":"915b8202-db44-4328-833d-8968478aff2e","Type":"ContainerStarted","Data":"e269d5bbc01240732ace6c81d85731045526c823f20235504f30b365713d4211"}
Sep 15 19:34:11.304547 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:11.304076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:11.321794 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:11.321732    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q" podStartSLOduration=0.903707668 podStartE2EDuration="3.321711516s" totalImagesPullingTime="2.418003848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 19:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:11.320002506 +0000 UTC m=+2603.474370975" watchObservedRunningTime="2026-09-15 19:34:11.321711516 +0000 UTC m=+2603.476079984"
Sep 15 19:34:17.316198 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:17.316160    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-l292q"
Sep 15 19:34:28.947570 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:28.947520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:34:29.039536 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:29.039499    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log" limit=1024
Sep 15 19:34:32.060151 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:32.060115    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jqw9v_53c11c0f-bbb6-49d8-896e-d17c292a1644/global-pull-secret-syncer/0.log"
Sep 15 19:34:32.108811 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:32.108781    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7zbgr_364d3196-120d-4ad1-b2b5-7c5d71b0351f/konnectivity-agent/0.log"
Sep 15 19:34:32.260435 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:32.260404    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-252.ec2.internal_b7e01db16d59f3dba97f5f5f5aaef31d/haproxy/0.log"
Sep 15 19:34:36.593601 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:36.593563    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-4c8xk_67c3cca0-8c9f-440f-8367-b8d10e9f3325/manager/0.log"
Sep 15 19:34:36.618943 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:36.618902    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vp5hc_5b8797b1-bc13-486c-bc6b-b65ddd358897/manager/0.log"
Sep 15 19:34:36.646857 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:36.646829    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7nplm_8dd9c656-57b0-4a2c-a51e-f7e6fbbcdf80/kuadrant-console-plugin/0.log"
Sep 15 19:34:36.694710 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:36.694674    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lbffm_61ad4629-6f4b-44f2-a96f-0bb7fc54d50a/manager/0.log"
Sep 15 19:34:36.774637 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:36.774580    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-r4dzv_a479125c-49b3-4e7c-9ab0-a679ae21906d/manager/0.log"
Sep 15 19:34:38.752073 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:38.752042    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rf469_a893bf4b-87b6-46e1-8e2e-01e94f891f11/node-exporter/0.log"
Sep 15 19:34:38.785766 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:38.785736    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rf469_a893bf4b-87b6-46e1-8e2e-01e94f891f11/kube-rbac-proxy/0.log"
Sep 15 19:34:38.812815 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:38.812791    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rf469_a893bf4b-87b6-46e1-8e2e-01e94f891f11/init-textfile/0.log"
Sep 15 19:34:38.839275 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:38.839249    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-96b78_503da95e-73a0-49a8-85e2-f1b38d92d85c/kube-rbac-proxy-main/0.log"
Sep 15 19:34:38.861142 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:38.861114    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-96b78_503da95e-73a0-49a8-85e2-f1b38d92d85c/kube-rbac-proxy-self/0.log"
Sep 15 19:34:38.882294 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:38.882249    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-96b78_503da95e-73a0-49a8-85e2-f1b38d92d85c/openshift-state-metrics/0.log"
Sep 15 19:34:39.137116 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:39.137083    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-ccdtf_c2655aa5-cd43-4859-936f-8cc57b374ffe/prometheus-operator/0.log"
Sep 15 19:34:39.160762 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:39.160726    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-ccdtf_c2655aa5-cd43-4859-936f-8cc57b374ffe/kube-rbac-proxy/0.log"
Sep 15 19:34:39.232544 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:39.232518    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cbd8d687-qs44c_45e2eef4-16ff-4a18-a686-479ab2c7d981/telemeter-client/0.log"
Sep 15 19:34:39.255806 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:39.255777    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cbd8d687-qs44c_45e2eef4-16ff-4a18-a686-479ab2c7d981/reload/0.log"
Sep 15 19:34:39.279638 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:39.279592    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cbd8d687-qs44c_45e2eef4-16ff-4a18-a686-479ab2c7d981/kube-rbac-proxy/0.log"
Sep 15 19:34:42.244957 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.244925    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64f57b995d-m9kwk_a3d86932-ad31-42cd-b23c-fead3b3f07e6/console/0.log"
Sep 15 19:34:42.292360 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.292315    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"]
Sep 15 19:34:42.297339 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.297308    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.299936 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.299909    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxzwt\"/\"kube-root-ca.crt\""
Sep 15 19:34:42.300797 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.300771    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xxzwt\"/\"default-dockercfg-7hxc4\""
Sep 15 19:34:42.301003 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.300978    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xxzwt\"/\"openshift-service-ca.crt\""
Sep 15 19:34:42.304454 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.304421    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"]
Sep 15 19:34:42.428461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.428412    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-podres\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.428461 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.428462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtmx\" (UniqueName: \"kubernetes.io/projected/2adbbae2-f3fc-4751-b501-cf3a9df15591-kube-api-access-bbtmx\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.428737 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.428562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-lib-modules\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.428737 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.428687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-proc\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.428819 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.428766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-sys\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-podres\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529588 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529598    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtmx\" (UniqueName: \"kubernetes.io/projected/2adbbae2-f3fc-4751-b501-cf3a9df15591-kube-api-access-bbtmx\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529836 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529690    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-lib-modules\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529836 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-proc\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529836 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529776    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-podres\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529988 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-sys\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529988 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529849    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-proc\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529988 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-lib-modules\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.529988 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.529913    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2adbbae2-f3fc-4751-b501-cf3a9df15591-sys\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.538288 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.538249    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtmx\" (UniqueName: \"kubernetes.io/projected/2adbbae2-f3fc-4751-b501-cf3a9df15591-kube-api-access-bbtmx\") pod \"perf-node-gather-daemonset-xcxk9\" (UID: \"2adbbae2-f3fc-4751-b501-cf3a9df15591\") " pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.609450 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.609397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:42.884556 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:42.884522    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"]
Sep 15 19:34:42.885821 ip-10-0-143-252 kubenswrapper[2584]: W0915 19:34:42.885787    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2adbbae2_f3fc_4751_b501_cf3a9df15591.slice/crio-eec73b2dede465404c6851f52752679e60bf0e89df6ccd489d1e23f648138f6c WatchSource:0}: Error finding container eec73b2dede465404c6851f52752679e60bf0e89df6ccd489d1e23f648138f6c: Status 404 returned error can't find the container with id eec73b2dede465404c6851f52752679e60bf0e89df6ccd489d1e23f648138f6c
Sep 15 19:34:43.507672 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:43.507628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9" event={"ID":"2adbbae2-f3fc-4751-b501-cf3a9df15591","Type":"ContainerStarted","Data":"248cfbdff689f9488d6842dbb596099ebc65448094f5fcab1798d89825ebbac4"}
Sep 15 19:34:43.508093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:43.507686    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9" event={"ID":"2adbbae2-f3fc-4751-b501-cf3a9df15591","Type":"ContainerStarted","Data":"eec73b2dede465404c6851f52752679e60bf0e89df6ccd489d1e23f648138f6c"}
Sep 15 19:34:43.508093 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:43.507713    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:43.524022 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:43.523964    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9" podStartSLOduration=1.523943637 podStartE2EDuration="1.523943637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 19:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 19:34:43.523270174 +0000 UTC m=+2635.677638643" watchObservedRunningTime="2026-09-15 19:34:43.523943637 +0000 UTC m=+2635.678312104"
Sep 15 19:34:44.318521 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:44.318486    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-58p5d_c098a940-98ba-4e64-a691-e2cd0d6a41bc/dns/0.log"
Sep 15 19:34:44.341950 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:44.341920    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-58p5d_c098a940-98ba-4e64-a691-e2cd0d6a41bc/kube-rbac-proxy/0.log"
Sep 15 19:34:44.547688 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:44.547663    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-px5bd_5fbf53cb-b32a-4e88-b5c1-2959df28a398/dns-node-resolver/0.log"
Sep 15 19:34:45.090752 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:45.090716    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8cdd6fd85-rh95h_8c2a232d-30d1-4941-af91-f2317ea2d488/registry/0.log"
Sep 15 19:34:45.140141 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:45.140095    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-khm77_9c258100-5235-4aee-b9a7-7e97267b0cb0/node-ca/0.log"
Sep 15 19:34:46.100630 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:46.100592    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62_ad422d98-f7bb-4f31-9101-a1a9446fdb25/istio-proxy/0.log"
Sep 15 19:34:46.515522 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:46.515494    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65488ff6c7-b8fc4_2a1612dd-403c-4e3b-a679-23748bd36b1a/kube-auth-proxy/0.log"
Sep 15 19:34:46.545387 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:46.545355    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zp9s8_0eb5d451-4682-4cf9-ac47-60323346ce64/istio-proxy/0.log"
Sep 15 19:34:46.547625 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:46.547595    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zp9s8_0eb5d451-4682-4cf9-ac47-60323346ce64/istio-proxy/1.log"
Sep 15 19:34:46.629678 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:46.629639    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-647fd6f876-2xxs8_042f4fd4-4470-4b60-927d-37c53e0f25d7/router/0.log"
Sep 15 19:34:47.523269 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:47.523244    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7jnzj_8a47024c-545b-4dc3-b464-77bc724c4d1d/serve-healthcheck-canary/0.log"
Sep 15 19:34:48.215816 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:48.215787    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v6g27_5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6/kube-rbac-proxy/0.log"
Sep 15 19:34:48.245017 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:48.244984    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v6g27_5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6/exporter/0.log"
Sep 15 19:34:48.266526 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:48.266479    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v6g27_5f37fd67-89d8-42d6-9ce9-cded8dc1f7f6/extractor/0.log"
Sep 15 19:34:49.149266 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:49.149236    2584 scope.go:117] "RemoveContainer" containerID="37e4a492bed17199fa44e2d5389e3141110653c751a5b54e67e72950441c1cd0"
Sep 15 19:34:49.158999 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:49.158935    2584 scope.go:117] "RemoveContainer" containerID="aaf25cfd5cb3db65cccef8bea30f59039aa47856e3c4eaa792b487ae7d50b570"
Sep 15 19:34:49.525355 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:49.525326    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xxzwt/perf-node-gather-daemonset-xcxk9"
Sep 15 19:34:50.395211 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:50.395182    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-h5m64_dcaa3581-8da6-4a75-8ca4-b453c866e980/manager/0.log"
Sep 15 19:34:50.620778 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:50.620742    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9g9t4_2444b57d-2402-4b84-962b-5d06909775d8/manager/0.log"
Sep 15 19:34:50.791069 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:50.791030    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sxr2k_b322b894-053a-4e46-8f24-2c72a523cf69/manager/0.log"
Sep 15 19:34:50.989443 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:50.989416    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d9df7495-t8z4v_6ab30393-65d5-4511-989c-826b66e83554/manager/0.log"
Sep 15 19:34:51.010977 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:51.010947    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d9df7495-t8z4v_6ab30393-65d5-4511-989c-826b66e83554/copy-manifests/0.log"
Sep 15 19:34:52.567072 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:52.567040    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gw8gw_0169b61f-90ff-45b2-a6fa-39021ee6ffff/openshift-lws-operator/0.log"
Sep 15 19:34:52.698685 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:52.698655    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62_ad422d98-f7bb-4f31-9101-a1a9446fdb25/istio-proxy/0.log"
Sep 15 19:34:53.080459 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:53.080429    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65488ff6c7-b8fc4_2a1612dd-403c-4e3b-a679-23748bd36b1a/kube-auth-proxy/0.log"
Sep 15 19:34:53.098529 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:53.098501    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zp9s8_0eb5d451-4682-4cf9-ac47-60323346ce64/istio-proxy/0.log"
Sep 15 19:34:53.100994 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:53.100964    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zp9s8_0eb5d451-4682-4cf9-ac47-60323346ce64/istio-proxy/1.log"
Sep 15 19:34:53.141931 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:53.141902    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-647fd6f876-2xxs8_042f4fd4-4470-4b60-927d-37c53e0f25d7/router/0.log"
Sep 15 19:34:54.582952 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:54.582920    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dgc62_ad422d98-f7bb-4f31-9101-a1a9446fdb25/istio-proxy/0.log"
Sep 15 19:34:54.980745 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:54.980668    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65488ff6c7-b8fc4_2a1612dd-403c-4e3b-a679-23748bd36b1a/kube-auth-proxy/0.log"
Sep 15 19:34:55.001626 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:55.001576    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zp9s8_0eb5d451-4682-4cf9-ac47-60323346ce64/istio-proxy/0.log"
Sep 15 19:34:55.004175 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:55.004154    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zp9s8_0eb5d451-4682-4cf9-ac47-60323346ce64/istio-proxy/1.log"
Sep 15 19:34:55.053567 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:55.053530    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-647fd6f876-2xxs8_042f4fd4-4470-4b60-927d-37c53e0f25d7/router/0.log"
Sep 15 19:34:58.059951 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:58.059898    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-x9rlg_de912687-0dfa-4dd1-8562-1e01213aee3f/kube-storage-version-migrator-operator/1.log"
Sep 15 19:34:58.060973 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:58.060954    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-x9rlg_de912687-0dfa-4dd1-8562-1e01213aee3f/kube-storage-version-migrator-operator/0.log"
Sep 15 19:34:59.347219 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.347187    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/kube-multus-additional-cni-plugins/0.log"
Sep 15 19:34:59.367941 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.367913    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/egress-router-binary-copy/0.log"
Sep 15 19:34:59.388254 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.388224    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/cni-plugins/0.log"
Sep 15 19:34:59.420893 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.420860    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/bond-cni-plugin/0.log"
Sep 15 19:34:59.440947 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.440920    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/routeoverride-cni/0.log"
Sep 15 19:34:59.461263 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.461233    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/whereabouts-cni-bincopy/0.log"
Sep 15 19:34:59.481730 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.481704    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ql8n6_9ad82e4b-7d28-409e-a9b3-0bb88ba2a914/whereabouts-cni/0.log"
Sep 15 19:34:59.572807 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.572769    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zjntw_cfc2c1a6-b49e-406c-a839-ca526167e28a/kube-multus/0.log"
Sep 15 19:34:59.687876 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.687785    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x77j7_7aff6fb0-49d5-4729-b743-fdf79154f869/network-metrics-daemon/0.log"
Sep 15 19:34:59.705726 ip-10-0-143-252 kubenswrapper[2584]: I0915 19:34:59.705700    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x77j7_7aff6fb0-49d5-4729-b743-fdf79154f869/kube-rbac-proxy/0.log"